Jul 17 21:45:36.480413 ip-10-0-129-11 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 21:45:36.927742 ip-10-0-129-11 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 21:45:36.927742 ip-10-0-129-11 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 21:45:36.927742 ip-10-0-129-11 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 21:45:36.927742 ip-10-0-129-11 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 21:45:36.927742 ip-10-0-129-11 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 21:45:36.928755 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.928630    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 21:45:36.935391 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935369    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:45:36.935391 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935391    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:45:36.935391 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935395    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935398    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935401    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935403    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935407    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935409    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935412    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935415    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935418    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935420    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935424    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935427    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935429    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935432    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935435    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935437    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935440    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935443    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935454    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935457    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:45:36.935494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935460    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935462    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935465    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935469    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935471    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935474    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935476    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935479    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935481    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935485    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935489    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935492    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935494    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935498    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935502    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935505    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935508    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935511    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935514    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:45:36.935966 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935517    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935520    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935522    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935525    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935527    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935529    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935532    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935534    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935537    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935539    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935541    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935544    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935547    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935549    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935552    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935555    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935558    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935561    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935563    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:45:36.936492 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935566    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935568    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935570    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935573    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935576    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935579    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935582    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935584    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935586    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935589    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935592    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935594    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935596    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935599    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935601    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935603    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935606    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935608    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935611    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935614    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:45:36.936945 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935618    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935621    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935623    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935626    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935628    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935631    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935633    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935636    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935640    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935643    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935645    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935648    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935650    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935653    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935655    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935658    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935660    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935663    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.935665    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936401    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:45:36.937450 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936407    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936410    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936413    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936416    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936419    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936422    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936425    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936427    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936430    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936433    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936436    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936438    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936441    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936443    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936446    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936448    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936451    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936453    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936455    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936459    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:45:36.937921 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936462    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936465    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936467    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936470    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936473    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936475    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936477    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936480    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936482    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936487    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936490    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936493    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936497    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936500    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936502    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936505    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936508    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936510    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936513    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:45:36.938494 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936516    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936518    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936521    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936523    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936526    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936528    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936531    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936533    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936536    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936538    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936542    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936545    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936548    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936914    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936918    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936921    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936925    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936928    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936931    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936934    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:45:36.939328 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936936    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936939    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936942    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936945    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936947    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936950    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936953    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936955    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936958    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936961    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936964    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936966    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936968    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936971    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936973    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936976    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936978    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936980    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936983    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936985    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:45:36.939818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936988    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936990    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936992    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936994    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.936997    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.937000    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.937004    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.937007    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.937009    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.937012    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.937014    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.937016    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.937019    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.937021    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.937024    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.937026    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.937029    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.937032    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.937034    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:45:36.940325 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938790    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938803    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938808    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938812    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938817    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938821    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938826    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938831    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938835    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938839    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938843    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938852    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938856    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938861    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938865    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938869    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938873    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938878    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938882    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938888    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:45:36.940779 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938892    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938896    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938903    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938908    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938917    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938921    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938928    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938934    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938939    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938944    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938949    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938954    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938958    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938963    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938967    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938971    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938981    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938985    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938990    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:45:36.941287 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938994    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.938998    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939004    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939008    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939012    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939017    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939021    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939025    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939034    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939039    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939043    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939047    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939052    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939057    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939061    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939082    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939087    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939091    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939095    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939099    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:45:36.941737 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939103    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939112    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939116    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939120    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939124    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939128    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939137    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939142    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939147    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939151    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939156    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939160    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939165    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939175    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939179    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939184    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939188    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939192    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939196    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939200    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:45:36.942247 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939204    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939208    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939213    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939217    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939222    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939226    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939236    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939240    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939245    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939249    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939253    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939258    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939262    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939266    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939270    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939274    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939278    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939282    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939510    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.939518    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:45:36.942750 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939615    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939624    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939632    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939637    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939641    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939645    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939649    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939654    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939657    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939660    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939664    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939667    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939670    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939673    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939676    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939679    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939682    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939685    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939690    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939693    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939697    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939700    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939703    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939707    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 21:45:36.943248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939710    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939713    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939716    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939719    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939722    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939724    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939728    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939731    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939736    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939739    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939742    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939745    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939748    2565 flags.go:64] FLAG: --enable-server="true"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939751    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939756    2565 flags.go:64] FLAG: --event-burst="100"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939758    2565 flags.go:64] FLAG: --event-qps="50"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939761    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939764    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939767    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939771    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939774    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939776    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939779    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939782    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939785    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 21:45:36.943840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939788    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939791    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939794    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939797    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939800    2565 flags.go:64] FLAG: --feature-gates=""
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939803    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939806    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939809    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939812    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939815    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939818    2565 flags.go:64] FLAG: --help="false"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939821    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-129-11.ec2.internal"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939824    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939828    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939831    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939835    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939839    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939842    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939845    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939848    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939851    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939854    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939857    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939860    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 17 21:45:36.944527 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939863    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939866    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939868    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939871    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939874    2565 flags.go:64] FLAG: --lock-file=""
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939877    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939879    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939882    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939888    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939890    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939893    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939896    2565 flags.go:64] FLAG: --logging-format="text"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939899    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939902    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939904    2565 flags.go:64] FLAG: --manifest-url=""
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939907    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939912    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939915    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939919    2565 flags.go:64] FLAG: --max-pods="110"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939922    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939925    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939929    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939932    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939935    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939938    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 21:45:36.945123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939941    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939949    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939952    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939955    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939958    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939961    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939966    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939968    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939971    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939974    2565 flags.go:64] FLAG: --port="10250"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939977    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939980    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e94fbb5e44dba54c"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939983    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939986    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939989    2565 flags.go:64] FLAG: --register-node="true"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939991    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939995    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.939998    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940001    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940004    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940008    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940011    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940013    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940016    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940019    2565 flags.go:64] FLAG: --runonce="false"
Jul 17 21:45:36.945724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940022    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940025    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940027    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940031    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940036    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940039    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940042    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940045    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940047    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940050    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940053    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940056    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940059    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940078    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940081    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940087    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940090    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940093    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940097    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940099    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940102    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940105    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940108    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940110    2565 flags.go:64] FLAG: --v="2"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940114    2565 flags.go:64] FLAG: --version="false"
Jul 17 21:45:36.946370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940118    2565 flags.go:64] FLAG: --vmodule=""
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940123    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.940126    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940247    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940250    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940254    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940258    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940262    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940266    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940268    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940272    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940275    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940278    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940281    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940283    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940286    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940288    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940291    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940294    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940297    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:45:36.946979 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940299    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940302    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940304    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940309    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940312    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940314    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940317    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940320    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940322    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940325    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940329    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940332    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940335    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940338    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940342    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940344    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940347    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940350    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:45:36.947564 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940352    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940355    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940357    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940360    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940362    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940366    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940369    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940371    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940374    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940376    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940378    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940381    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940383    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940385    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940388    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940390    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940393    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940395    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940397    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940400    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:45:36.948006 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940402    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940405    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940407    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940409    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940412    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940414    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940417    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940419    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940421    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940425    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940427    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940430    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940432    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940435    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940437    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940439    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940442    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940445    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940448    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940450    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:45:36.948519 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940453    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940455    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940457    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940460    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940462    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940464    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940467    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940469    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940472    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940474    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940477    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940480    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940482    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940485    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940487    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940489    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940492    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940495    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940497    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940499    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:45:36.949002 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940502    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:45:36.949500 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940504    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:45:36.949500 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940507    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:45:36.949500 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.940509    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:45:36.949500 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.941252    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 21:45:36.949500 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.948910    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 21:45:36.949500 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.948928    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 21:45:36.949500 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.948941    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 21:45:36.949500 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.948946    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 21:45:36.949500 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949007    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:45:36.949500 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949012    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:45:36.949500 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949015    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:45:36.949500 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949019    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:45:36.949500 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949022    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:45:36.949500 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949025    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:45:36.949500 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949028    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949031    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949034    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949037    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949040    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949043    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949046    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949048    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949051    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949054    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949056    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949059    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949061    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949083    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949086    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949090    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949094    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949097    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949100    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:45:36.949908 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949108    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949111    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949114    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949117    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949120    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949123    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949126    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949129    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949132    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949134    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949137    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949139    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949142    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949144    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949147    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949149    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949152    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949155    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949157    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949159    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:45:36.950389 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949162    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949164    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949167    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949169    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949172    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949174    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949177    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949179    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949182    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949186    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949188    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949191    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949193    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949197    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949199    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949202    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949205    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949208    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949211    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949213    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:45:36.950885 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949216    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949218    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949221    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949223    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949226    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949228    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949230    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949233    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949235    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949238    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949240    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949242    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949245    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949248    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949251    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949255    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949257    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949260    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949263    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949266    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:45:36.951495 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949268    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:45:36.951967 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949271    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:45:36.951967 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949275    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:45:36.951967 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949278    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:45:36.951967 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949282    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:45:36.951967 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949284    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:45:36.951967 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949288    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:45:36.951967 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949290    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:45:36.951967 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949293    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:45:36.951967 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949295    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:45:36.951967 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949298    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:45:36.951967 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949300    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:45:36.951967 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949302    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:45:36.951967 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949305    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:45:36.951967 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.949310    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949423    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949427    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949430    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949433    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949436    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949438    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949441    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949444    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949447    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949449    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949452    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949455    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949457    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949459    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949462    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949464    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949467    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949469    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949472    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:45:36.952337 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949475    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949477    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949480    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949482    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949484    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949488    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949490    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949493    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949496    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949498    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949500    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949503    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949506    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949508    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949511    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949513    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949516    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949518    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949521    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949523    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:45:36.952818 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949525    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949538    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949541    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949544    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949547    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949549    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949552    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949554    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949557    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949559    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949562    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949565    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949567    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949570    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949572    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949575    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949578    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949580    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949584    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949587    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:45:36.953302 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949589    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949592    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949594    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949597    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949599    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949601    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949605    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949609    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949612    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949614    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949617    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949620    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949622    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949625    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949627    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949629    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949632    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949634    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949637    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949639    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:45:36.953809 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949641    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949644    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949648    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949652    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949655    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949657    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949661    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949664    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949667    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949669    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949672    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949676    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949679    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949681    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949685    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949687    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949689    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949692    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949694    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:45:36.954340 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:36.949697    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:45:36.954803 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.949702    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 21:45:36.954803 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.949824    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 21:45:36.954803 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.952291    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 21:45:36.954803 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.953915    2565 server.go:1019] "Starting client certificate rotation"
Jul 17 21:45:36.954803 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.954026    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 21:45:36.954959 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.954896    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 21:45:36.956723 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.956710    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 21:45:36.957896 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.957879    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 21:45:36.971294 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.971249    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 17 21:45:36.974156 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.974139    2565 log.go:25] "Validated CRI v1 image API"
Jul 17 21:45:36.975056 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.975040    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 21:45:36.980962 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.980941    2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b5ee03ac-722c-4bdd-971c-888894ce8197:/dev/nvme0n1p3 c26f8ec2-4d5e-4282-934b-e10427e27410:/dev/nvme0n1p4]
Jul 17 21:45:36.981014 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.980962    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 21:45:36.985545 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.985525    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 21:45:36.986729 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.986621    2565 manager.go:216] Machine: {Timestamp:2026-07-17 21:45:36.985020344 +0000 UTC m=+0.374036986 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106353 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29de215ad819fae75d34ad4eb1d8a4 SystemUUID:ec29de21-5ad8-19fa-e75d-34ad4eb1d8a4 BootID:00a92dbd-6387-4a47-af20-3be7791c716b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:67:ef:2b:ea:cf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:67:ef:2b:ea:cf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:e9:69:37:ee:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 21:45:36.986729 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.986729    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 21:45:36.986833 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.986819    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 21:45:36.990275 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.990247    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 21:45:36.990412 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.990279    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-11.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 21:45:36.991115 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.991104    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 21:45:36.991157 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.991118    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 21:45:36.991157 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.991126    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 21:45:36.991157 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.991140    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 21:45:36.993059 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.993048    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 21:45:36.997734 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.997720    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 21:45:36.997877 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.997867    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 21:45:36.997916 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.997904    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 21:45:36.997948 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.997919    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 21:45:36.997948 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.997936    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 21:45:36.997948 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.997945    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 17 21:45:36.998042 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.997956    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 21:45:36.998360 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.998348    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 21:45:36.998406 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:36.998366    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 21:45:37.001734 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.001717    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 21:45:37.003344 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.003325    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g4w8z"
Jul 17 21:45:37.004242 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.004228    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 21:45:37.004295 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.004259    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 21:45:37.005769 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.005752    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 21:45:37.005841 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.005777    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 21:45:37.005841 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.005787    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 21:45:37.005841 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.005800    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 21:45:37.005841 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.005821    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 21:45:37.005841 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.005832    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 21:45:37.005841 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.005842    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 21:45:37.005995 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.005848    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 21:45:37.005995 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.005856    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 21:45:37.005995 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.005862    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 21:45:37.005995 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.005871    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 21:45:37.005995 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.005877    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 21:45:37.006834 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.006820    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 21:45:37.006834 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.006834    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 21:45:37.007372 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.007339    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 21:45:37.007462 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.007416    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-11.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 21:45:37.008731 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.008705    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g4w8z"
Jul 17 21:45:37.010842 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.010829    2565 server.go:1268] "Started kubelet"
Jul 17 21:45:37.010973 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.010937    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 21:45:37.011045 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.011004    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 21:45:37.011457 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.011424    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 21:45:37.011857 ip-10-0-129-11 systemd[1]: Started Kubernetes Kubelet.
Jul 17 21:45:37.015840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.015818    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 21:45:37.017683 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.017662    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 21:45:37.019704 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.019669    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 21:45:37.025925 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.025906    2565 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-11.ec2.internal" not found
Jul 17 21:45:37.030110 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.030081    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 21:45:37.030110 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.030091    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 21:45:37.030254 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.030228    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-11.ec2.internal\" not found"
Jul 17 21:45:37.031028 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.030934    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 21:45:37.031412 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.031125    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 21:45:37.031412 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.031147    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 21:45:37.031412 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.031293    2565 factory.go:55] Registering systemd factory
Jul 17 21:45:37.031412 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.031306    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 21:45:37.031412 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.031316    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 21:45:37.031412 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.031310    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 17 21:45:37.031715 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.031538    2565 factory.go:153] Registering CRI-O factory
Jul 17 21:45:37.031715 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.031552    2565 factory.go:223] Registration of the crio container factory successfully
Jul 17 21:45:37.031715 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.031611    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 21:45:37.031715 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.031642    2565 factory.go:103] Registering Raw factory
Jul 17 21:45:37.031715 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.031657    2565 manager.go:1196] Started watching for new ooms in manager
Jul 17 21:45:37.032414 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.032397    2565 manager.go:349] Starting recovery of all containers
Jul 17 21:45:37.032844 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.032823    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:45:37.035251 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.035223    2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-11.ec2.internal\" not found" node="ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.040319 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.040300    2565 manager.go:354] Recovery completed
Jul 17 21:45:37.044419 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.044402    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:45:37.045476 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.045462    2565 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-11.ec2.internal" not found
Jul 17 21:45:37.047058 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.047039    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:45:37.047144 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.047103    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:45:37.047144 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.047120    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:45:37.048224 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.048200    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 21:45:37.048224 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.048224    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 21:45:37.048351 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.048252    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 21:45:37.049524 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.049508    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:45:37.050459 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.050446    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:45:37.050538 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.050472    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:45:37.050538 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.050482    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:45:37.050538 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.050505    2565 policy_none.go:49] "None policy: Start"
Jul 17 21:45:37.050538 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.050523    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 21:45:37.050720 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.050539    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 21:45:37.052653 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.052640    2565 policy_none.go:47] "Start"
Jul 17 21:45:37.111551 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.097561    2565 manager.go:337] "Starting Device Plugin manager"
Jul 17 21:45:37.111551 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.097760    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 21:45:37.111551 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.097775    2565 server.go:87] "Starting device plugin registration server"
Jul 17 21:45:37.111551 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.098026    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 21:45:37.111551 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.098036    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 21:45:37.111551 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.098121    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 21:45:37.111551 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.098210    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 21:45:37.111551 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.098219    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 21:45:37.111551 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.098780    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 21:45:37.111551 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.098831    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-11.ec2.internal\" not found"
Jul 17 21:45:37.111551 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.104301    2565 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-129-11.ec2.internal" not found
Jul 17 21:45:37.174447 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.174407    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 21:45:37.175775 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.175752    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 21:45:37.175897 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.175803    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 21:45:37.175897 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.175826    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 21:45:37.175897 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.175859    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 21:45:37.178442 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.178383    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:45:37.198550 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.198515    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:45:37.199470 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.199454    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:45:37.199561 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.199484    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:45:37.199561 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.199493    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:45:37.199561 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.199517    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.208578 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.208559    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.208664 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.208585    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-11.ec2.internal\": node \"ip-10-0-129-11.ec2.internal\" not found"
Jul 17 21:45:37.218982 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.218958    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-11.ec2.internal\" not found"
Jul 17 21:45:37.276575 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.276528    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-11.ec2.internal"]
Jul 17 21:45:37.276687 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.276619    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:45:37.277608 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.277592    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:45:37.277679 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.277626    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:45:37.277679 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.277635    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:45:37.279968 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.279955    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:45:37.280102 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.280089    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.280142 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.280117    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:45:37.280716 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.280701    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:45:37.280776 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.280723    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:45:37.280776 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.280734    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:45:37.280776 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.280765    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:45:37.280882 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.280782    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:45:37.280882 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.280791    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:45:37.283002 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.282985    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.283112 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.283008    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:45:37.283809 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.283792    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:45:37.283878 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.283824    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:45:37.283878 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.283839    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:45:37.314967 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.314940    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-11.ec2.internal\" not found" node="ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.319326 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.319311    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-11.ec2.internal\" not found"
Jul 17 21:45:37.319481 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.319467    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-11.ec2.internal\" not found" node="ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.332732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.332700    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/957fcdbc1554bfacb8cd71b42c384138-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal\" (UID: \"957fcdbc1554bfacb8cd71b42c384138\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.332732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.332730    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/957fcdbc1554bfacb8cd71b42c384138-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal\" (UID: \"957fcdbc1554bfacb8cd71b42c384138\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.332906 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.332749    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f1502cc83beae535f93cdbb53034f5e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-11.ec2.internal\" (UID: \"1f1502cc83beae535f93cdbb53034f5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.420040 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.419989    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-11.ec2.internal\" not found"
Jul 17 21:45:37.433405 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.433345    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/957fcdbc1554bfacb8cd71b42c384138-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal\" (UID: \"957fcdbc1554bfacb8cd71b42c384138\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.433405 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.433376    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/957fcdbc1554bfacb8cd71b42c384138-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal\" (UID: \"957fcdbc1554bfacb8cd71b42c384138\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.433405 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.433394    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f1502cc83beae535f93cdbb53034f5e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-11.ec2.internal\" (UID: \"1f1502cc83beae535f93cdbb53034f5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.433549 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.433435    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f1502cc83beae535f93cdbb53034f5e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-11.ec2.internal\" (UID: \"1f1502cc83beae535f93cdbb53034f5e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.433549 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.433454    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/957fcdbc1554bfacb8cd71b42c384138-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal\" (UID: \"957fcdbc1554bfacb8cd71b42c384138\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.433549 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.433471    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/957fcdbc1554bfacb8cd71b42c384138-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal\" (UID: \"957fcdbc1554bfacb8cd71b42c384138\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.520810 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.520773    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-11.ec2.internal\" not found"
Jul 17 21:45:37.618354 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.618331    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.621482 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.621462    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-11.ec2.internal\" not found"
Jul 17 21:45:37.623353 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.623156    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.722221 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.722126    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-11.ec2.internal\" not found"
Jul 17 21:45:37.822654 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.822622    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-11.ec2.internal\" not found"
Jul 17 21:45:37.879730 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.879705    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:45:37.930928 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.930900    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.942920 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.942901    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 21:45:37.944417 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.944403    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.953060 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.953039    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 21:45:37.953200 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.953174    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 21:45:37.953200 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.953179    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 21:45:37.953281 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.953178    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 21:45:37.953281 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:37.953215    2565 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a9afe288caea44018a0953c2a21408ee-53e4f526dde1cf28.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.129.11:43560->54.225.168.157:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-11.ec2.internal"
Jul 17 21:45:37.998531 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:37.998506    2565 apiserver.go:52] "Watching apiserver"
Jul 17 21:45:38.017106 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.017061    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 21:45:38.018004 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.017982    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-hf7sc","kube-system/konnectivity-agent-5xm2r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal","openshift-multus/multus-p2d44","openshift-multus/network-metrics-daemon-cd95t","openshift-network-diagnostics/network-check-target-mg8bv","openshift-network-operator/iptables-alerter-qnscl","kube-system/kube-apiserver-proxy-ip-10-0-129-11.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm","openshift-cluster-node-tuning-operator/tuned-6mrs7","openshift-dns/node-resolver-sq8rg","openshift-image-registry/node-ca-ls9f6","openshift-multus/multus-additional-cni-plugins-h2lwh","openshift-ovn-kubernetes/ovnkube-node-bhftj"]
Jul 17 21:45:38.018131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.017980    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 21:40:37 +0000 UTC" deadline="2028-02-29 19:53:47.973169975 +0000 UTC"
Jul 17 21:45:38.018131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.018025    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14206h8m9.955146828s"
Jul 17 21:45:38.020429 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.020402    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qnscl"
Jul 17 21:45:38.022938 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.022915    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sc5s8\""
Jul 17 21:45:38.023107 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.023087    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:45:38.023187 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.023087    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 21:45:38.023187 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.023096    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 21:45:38.023280 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.023194    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.026215 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.025713    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.026558 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.026535    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 21:45:38.026663 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.026569    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 21:45:38.026663 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.026607    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 21:45:38.026663 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.026538    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 21:45:38.026820 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.026766    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 21:45:38.026820 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.026774    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fd6sd\""
Jul 17 21:45:38.027724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.027706    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.027880 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.027866    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9brjf\""
Jul 17 21:45:38.028045 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.028029    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 21:45:38.028138 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.028111    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 21:45:38.028138 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.028111    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 21:45:38.029107 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.029092    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 21:45:38.029167 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.029109    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 21:45:38.029200 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.029118    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 21:45:38.029932 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.029918    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cjbwk\""
Jul 17 21:45:38.030001 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.029943    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 21:45:38.030052 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.030032    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:38.030116 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.030102    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:45:38.030247 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.030234    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 21:45:38.032105 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.032089    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:38.032216 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.032139    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:45:38.034393 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.034375    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.036418 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036400    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-run-openvswitch\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.036506 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036424    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-node-log\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.036506 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036440    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-ovnkube-script-lib\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.036506 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036457    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f619dbf4-d977-437e-9c8c-bd7eaf044562-iptables-alerter-script\") pod \"iptables-alerter-qnscl\" (UID: \"f619dbf4-d977-437e-9c8c-bd7eaf044562\") " pod="openshift-network-operator/iptables-alerter-qnscl"
Jul 17 21:45:38.036506 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036483    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d604d7d6-843e-47f3-b837-31545356dbf3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.036708 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036524    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-var-lib-openvswitch\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.036708 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036575    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-etc-openvswitch\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.036708 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036588    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a05b7f74-b162-43d1-b600-53c5904268de-cni-binary-copy\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.036708 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-run-netns\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.036708 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036623    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-var-lib-kubelet\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.036708 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036646    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch62k\" (UniqueName: \"kubernetes.io/projected/a05b7f74-b162-43d1-b600-53c5904268de-kube-api-access-ch62k\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.036708 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036678    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f619dbf4-d977-437e-9c8c-bd7eaf044562-host-slash\") pod \"iptables-alerter-qnscl\" (UID: \"f619dbf4-d977-437e-9c8c-bd7eaf044562\") " pod="openshift-network-operator/iptables-alerter-qnscl"
Jul 17 21:45:38.037027 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036724    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-ovnkube-config\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.037027 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036737    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 21:45:38.037027 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036753    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-ovn-node-metrics-cert\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.037027 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-multus-conf-dir\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.037027 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d604d7d6-843e-47f3-b837-31545356dbf3-cnibin\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.037027 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036852    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d604d7d6-843e-47f3-b837-31545356dbf3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.037027 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036876    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrjxr\" (UniqueName: \"kubernetes.io/projected/d604d7d6-843e-47f3-b837-31545356dbf3-kube-api-access-qrjxr\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.037027 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036907    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.037027 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036952    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-var-lib-cni-multus\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.037027 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs\") pod \"network-metrics-daemon-cd95t\" (UID: \"ab648b99-a119-46ea-894c-2c83d2658261\") " pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:38.037027 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.036981    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 21:45:38.037027 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037016    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d604d7d6-843e-47f3-b837-31545356dbf3-os-release\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037040    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-multus-cni-dir\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037080    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-run-systemd\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037104    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-run-ovn\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037143    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-769p8\""
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037150    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-cni-bin\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037198    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a05b7f74-b162-43d1-b600-53c5904268de-multus-daemon-config\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037281    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037283    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x456\" (UniqueName: \"kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456\") pod \"network-check-target-mg8bv\" (UID: \"7e18d002-d4f9-49ff-b425-fc3c56585a80\") " pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037406    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlqxh\" (UniqueName: \"kubernetes.io/projected/f619dbf4-d977-437e-9c8c-bd7eaf044562-kube-api-access-nlqxh\") pod \"iptables-alerter-qnscl\" (UID: \"f619dbf4-d977-437e-9c8c-bd7eaf044562\") " pod="openshift-network-operator/iptables-alerter-qnscl"
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-kubelet\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037445    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-systemd-units\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037451    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037468    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-slash\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037487    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-run-netns\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037501    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-log-socket\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-cni-netd\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.037595 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.037515    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:45:38.038732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjwms\" (UniqueName: \"kubernetes.io/projected/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-kube-api-access-bjwms\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.038732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-os-release\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.038732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037576    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-cnibin\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.038732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037593    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-multus-socket-dir-parent\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.038732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037614    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-var-lib-cni-bin\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.038732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037628    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-hostroot\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.038732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037641    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-run-multus-certs\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.038732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037682    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj2xh\" (UniqueName: \"kubernetes.io/projected/ab648b99-a119-46ea-894c-2c83d2658261-kube-api-access-tj2xh\") pod \"network-metrics-daemon-cd95t\" (UID: \"ab648b99-a119-46ea-894c-2c83d2658261\") " pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:38.038732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d604d7d6-843e-47f3-b837-31545356dbf3-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.038732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037723    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d604d7d6-843e-47f3-b837-31545356dbf3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.038732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037745    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-env-overrides\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.038732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037766    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-system-cni-dir\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.038732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037780    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-run-k8s-cni-cncf-io\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.038732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037799    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-etc-kubernetes\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.038732 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.037841    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d604d7d6-843e-47f3-b837-31545356dbf3-system-cni-dir\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.039338 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.039300    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5xm2r"
Jul 17 21:45:38.041471 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.041444    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 21:45:38.041562 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.041538    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p6hkn\""
Jul 17 21:45:38.041622 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.041584    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.042284 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.042146    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 21:45:38.042284 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.042192    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 21:45:38.044038 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.044013    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:45:38.044287 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.044271    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 21:45:38.044791 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.044776    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pxs2l\""
Jul 17 21:45:38.045390 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.045365    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sq8rg"
Jul 17 21:45:38.047549 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.047535    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 21:45:38.047625 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.047612    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sm6md\""
Jul 17 21:45:38.047792 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.047780    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 21:45:38.048037 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.048022    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ls9f6"
Jul 17 21:45:38.050876 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.050855    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 21:45:38.050996 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.050876    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vcsfw\""
Jul 17 21:45:38.050996 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.050910    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 21:45:38.050996 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.050861    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 21:45:38.051228 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.051212    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:45:38.069591 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.069571    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4zg5q"
Jul 17 21:45:38.077232 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.077205    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4zg5q"
Jul 17 21:45:38.131577 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.131554    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 21:45:38.138326 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a05b7f74-b162-43d1-b600-53c5904268de-multus-daemon-config\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.138445 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138345    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d604d7d6-843e-47f3-b837-31545356dbf3-system-cni-dir\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.138445 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138375    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-modprobe-d\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.138445 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138397    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-tuned\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.138445 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-kubelet\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.138445 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138440    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d604d7d6-843e-47f3-b837-31545356dbf3-system-cni-dir\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.138643 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-systemd-units\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.138643 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138477    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-kubelet\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.138643 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138481    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-systemd-units\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.138643 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138482    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-slash\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.138643 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138509    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-slash\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.138643 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-run-netns\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.138643 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138563    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-hostroot\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.138643 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138590    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-sys-fs\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.138643 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138618    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-run-netns\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.138643 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138630    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-hostroot\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.139199 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138619    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret\") pod \"global-pull-secret-syncer-hf7sc\" (UID: \"aeb396b3-2508-4c93-aeee-188c1b39fa9d\") " pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:38.139199 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.138873    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a05b7f74-b162-43d1-b600-53c5904268de-multus-daemon-config\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.139394 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139246    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-var-lib-cni-bin\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.139394 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d604d7d6-843e-47f3-b837-31545356dbf3-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.139394 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139339    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-var-lib-cni-bin\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.139521 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139391    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-sysctl-d\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.139521 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139423    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-run\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.139521 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c3a69f3-bd78-488e-8f56-8f26361e5165-tmp\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.139521 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139487    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b77bccab-4a6d-4e18-8206-9ad86e070317-host\") pod \"node-ca-ls9f6\" (UID: \"b77bccab-4a6d-4e18-8206-9ad86e070317\") " pod="openshift-image-registry/node-ca-ls9f6"
Jul 17 21:45:38.139719 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-system-cni-dir\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.139719 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139647    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-system-cni-dir\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.139719 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139655    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-etc-kubernetes\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.139719 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139680    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-sysconfig\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.139904 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139706    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-run-openvswitch\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.139904 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139718    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-etc-kubernetes\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.139904 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-node-log\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.139904 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139768    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-ovnkube-script-lib\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.139904 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139769    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-run-openvswitch\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.139904 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-node-log\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.139904 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139803    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f619dbf4-d977-437e-9c8c-bd7eaf044562-iptables-alerter-script\") pod \"iptables-alerter-qnscl\" (UID: \"f619dbf4-d977-437e-9c8c-bd7eaf044562\") " pod="openshift-network-operator/iptables-alerter-qnscl"
Jul 17 21:45:38.139904 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139831    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-etc-openvswitch\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.139904 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-run-netns\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.139904 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139866    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-var-lib-kubelet\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.139904 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch62k\" (UniqueName: \"kubernetes.io/projected/a05b7f74-b162-43d1-b600-53c5904268de-kube-api-access-ch62k\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.139904 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139862    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d604d7d6-843e-47f3-b837-31545356dbf3-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.140476 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139905    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-etc-openvswitch\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.140476 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79e9c71a-027c-4087-a6e0-1ecf5753a7b5-tmp-dir\") pod \"node-resolver-sq8rg\" (UID: \"79e9c71a-027c-4087-a6e0-1ecf5753a7b5\") " pod="openshift-dns/node-resolver-sq8rg"
Jul 17 21:45:38.140476 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139930    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-run-netns\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.140476 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.139991    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-var-lib-kubelet\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.140476 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140049    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-device-dir\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.140476 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140195    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aeb396b3-2508-4c93-aeee-188c1b39fa9d-kubelet-config\") pod \"global-pull-secret-syncer-hf7sc\" (UID: \"aeb396b3-2508-4c93-aeee-188c1b39fa9d\") " pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:38.140476 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-multus-conf-dir\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.140476 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140304    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d604d7d6-843e-47f3-b837-31545356dbf3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.140476 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140326    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdv96\" (UniqueName: \"kubernetes.io/projected/79e9c71a-027c-4087-a6e0-1ecf5753a7b5-kube-api-access-rdv96\") pod \"node-resolver-sq8rg\" (UID: \"79e9c71a-027c-4087-a6e0-1ecf5753a7b5\") " pod="openshift-dns/node-resolver-sq8rg"
Jul 17 21:45:38.140476 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140368    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-multus-conf-dir\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.140476 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjxgd\" (UniqueName: \"kubernetes.io/projected/b77bccab-4a6d-4e18-8206-9ad86e070317-kube-api-access-zjxgd\") pod \"node-ca-ls9f6\" (UID: \"b77bccab-4a6d-4e18-8206-9ad86e070317\") " pod="openshift-image-registry/node-ca-ls9f6"
Jul 17 21:45:38.140476 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aeb396b3-2508-4c93-aeee-188c1b39fa9d-dbus\") pod \"global-pull-secret-syncer-hf7sc\" (UID: \"aeb396b3-2508-4c93-aeee-188c1b39fa9d\") " pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:38.140476 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140447    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00ccd718-0d99-4948-84b4-6d0aaff1598b-konnectivity-ca\") pod \"konnectivity-agent-5xm2r\" (UID: \"00ccd718-0d99-4948-84b4-6d0aaff1598b\") " pod="kube-system/konnectivity-agent-5xm2r"
Jul 17 21:45:38.140476 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140450    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f619dbf4-d977-437e-9c8c-bd7eaf044562-iptables-alerter-script\") pod \"iptables-alerter-qnscl\" (UID: \"f619dbf4-d977-437e-9c8c-bd7eaf044562\") " pod="openshift-network-operator/iptables-alerter-qnscl"
Jul 17 21:45:38.140476 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140465    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-multus-cni-dir\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140515    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-run-systemd\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140521    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140543    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-run-ovn\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140574    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x456\" (UniqueName: \"kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456\") pod \"network-check-target-mg8bv\" (UID: \"7e18d002-d4f9-49ff-b425-fc3c56585a80\") " pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140580    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-run-systemd\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlqxh\" (UniqueName: \"kubernetes.io/projected/f619dbf4-d977-437e-9c8c-bd7eaf044562-kube-api-access-nlqxh\") pod \"iptables-alerter-qnscl\" (UID: \"f619dbf4-d977-437e-9c8c-bd7eaf044562\") " pod="openshift-network-operator/iptables-alerter-qnscl"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140617    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-run-ovn\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-sys\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140542    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-ovnkube-script-lib\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140657    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00ccd718-0d99-4948-84b4-6d0aaff1598b-agent-certs\") pod \"konnectivity-agent-5xm2r\" (UID: \"00ccd718-0d99-4948-84b4-6d0aaff1598b\") " pod="kube-system/konnectivity-agent-5xm2r"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140682    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-log-socket\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-cni-netd\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140716    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-multus-cni-dir\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjwms\" (UniqueName: \"kubernetes.io/projected/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-kube-api-access-bjwms\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140767    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-log-socket\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.141085 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140804    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-os-release\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140804    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d604d7d6-843e-47f3-b837-31545356dbf3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140832    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-multus-socket-dir-parent\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d604d7d6-843e-47f3-b837-31545356dbf3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140973    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-lib-modules\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140943    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-cni-netd\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.140985    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-os-release\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-multus-socket-dir-parent\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-cnibin\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-run-multus-certs\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141137    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj2xh\" (UniqueName: \"kubernetes.io/projected/ab648b99-a119-46ea-894c-2c83d2658261-kube-api-access-tj2xh\") pod \"network-metrics-daemon-cd95t\" (UID: \"ab648b99-a119-46ea-894c-2c83d2658261\") " pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141158    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-cnibin\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-kubernetes\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141189    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-run-multus-certs\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141195    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-host\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141227    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx7wg\" (UniqueName: \"kubernetes.io/projected/1c3a69f3-bd78-488e-8f56-8f26361e5165-kube-api-access-zx7wg\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141251    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-env-overrides\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.142006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141275    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-run-k8s-cni-cncf-io\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141297    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-registration-dir\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141336    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-run-k8s-cni-cncf-io\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141347    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d604d7d6-843e-47f3-b837-31545356dbf3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141399    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b77bccab-4a6d-4e18-8206-9ad86e070317-serviceca\") pod \"node-ca-ls9f6\" (UID: \"b77bccab-4a6d-4e18-8206-9ad86e070317\") " pod="openshift-image-registry/node-ca-ls9f6"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141424    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-socket-dir\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141448    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8q86\" (UniqueName: \"kubernetes.io/projected/14e20aba-c31d-4e1d-8a83-9ff790ede00a-kube-api-access-q8q86\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141482    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-var-lib-openvswitch\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a05b7f74-b162-43d1-b600-53c5904268de-cni-binary-copy\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141526    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d604d7d6-843e-47f3-b837-31545356dbf3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f619dbf4-d977-437e-9c8c-bd7eaf044562-host-slash\") pod \"iptables-alerter-qnscl\" (UID: \"f619dbf4-d977-437e-9c8c-bd7eaf044562\") " pod="openshift-network-operator/iptables-alerter-qnscl"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141570    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f619dbf4-d977-437e-9c8c-bd7eaf044562-host-slash\") pod \"iptables-alerter-qnscl\" (UID: \"f619dbf4-d977-437e-9c8c-bd7eaf044562\") " pod="openshift-network-operator/iptables-alerter-qnscl"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141571    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-var-lib-openvswitch\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141531    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d604d7d6-843e-47f3-b837-31545356dbf3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141596    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-sysctl-conf\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141621    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/79e9c71a-027c-4087-a6e0-1ecf5753a7b5-hosts-file\") pod \"node-resolver-sq8rg\" (UID: \"79e9c71a-027c-4087-a6e0-1ecf5753a7b5\") " pod="openshift-dns/node-resolver-sq8rg"
Jul 17 21:45:38.142646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141630    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-env-overrides\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141645    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-ovnkube-config\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141792    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-ovn-node-metrics-cert\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141817    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d604d7d6-843e-47f3-b837-31545356dbf3-cnibin\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141840    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrjxr\" (UniqueName: \"kubernetes.io/projected/d604d7d6-843e-47f3-b837-31545356dbf3-kube-api-access-qrjxr\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141909    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a05b7f74-b162-43d1-b600-53c5904268de-cni-binary-copy\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141911    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d604d7d6-843e-47f3-b837-31545356dbf3-cnibin\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-var-lib-cni-multus\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs\") pod \"network-metrics-daemon-cd95t\" (UID: \"ab648b99-a119-46ea-894c-2c83d2658261\") " pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.141971    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d604d7d6-843e-47f3-b837-31545356dbf3-os-release\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.142039    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a05b7f74-b162-43d1-b600-53c5904268de-host-var-lib-cni-multus\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.142089    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d604d7d6-843e-47f3-b837-31545356dbf3-os-release\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.142094    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-systemd\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.142170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-var-lib-kubelet\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.142139    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.142203    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:45:38.143131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.142211    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.143678 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.142296    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs podName:ab648b99-a119-46ea-894c-2c83d2658261 nodeName:}" failed. No retries permitted until 2026-07-17 21:45:38.642262414 +0000 UTC m=+2.031279060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs") pod "network-metrics-daemon-cd95t" (UID: "ab648b99-a119-46ea-894c-2c83d2658261") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:45:38.143678 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.142299    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.143678 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.142303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-ovnkube-config\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.143678 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.142397    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-cni-bin\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.143678 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.142237    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-host-cni-bin\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.145152 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.145132    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-ovn-node-metrics-cert\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.151922 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.151897    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:45:38.152051 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.151928    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:45:38.152051 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.151943    2565 projected.go:196] Error preparing data for projected volume kube-api-access-2x456 for pod openshift-network-diagnostics/network-check-target-mg8bv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:45:38.152051 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.152007    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456 podName:7e18d002-d4f9-49ff-b425-fc3c56585a80 nodeName:}" failed. No retries permitted until 2026-07-17 21:45:38.651993455 +0000 UTC m=+2.041010096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2x456" (UniqueName: "kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456") pod "network-check-target-mg8bv" (UID: "7e18d002-d4f9-49ff-b425-fc3c56585a80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:45:38.152511 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.152494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch62k\" (UniqueName: \"kubernetes.io/projected/a05b7f74-b162-43d1-b600-53c5904268de-kube-api-access-ch62k\") pod \"multus-p2d44\" (UID: \"a05b7f74-b162-43d1-b600-53c5904268de\") " pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.154895 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.154873    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjwms\" (UniqueName: \"kubernetes.io/projected/4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6-kube-api-access-bjwms\") pod \"ovnkube-node-bhftj\" (UID: \"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.155169 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.155141    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlqxh\" (UniqueName: \"kubernetes.io/projected/f619dbf4-d977-437e-9c8c-bd7eaf044562-kube-api-access-nlqxh\") pod \"iptables-alerter-qnscl\" (UID: \"f619dbf4-d977-437e-9c8c-bd7eaf044562\") " pod="openshift-network-operator/iptables-alerter-qnscl"
Jul 17 21:45:38.155280 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.155185    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj2xh\" (UniqueName: \"kubernetes.io/projected/ab648b99-a119-46ea-894c-2c83d2658261-kube-api-access-tj2xh\") pod \"network-metrics-daemon-cd95t\" (UID: \"ab648b99-a119-46ea-894c-2c83d2658261\") " pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:38.155344 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.155296    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrjxr\" (UniqueName: \"kubernetes.io/projected/d604d7d6-843e-47f3-b837-31545356dbf3-kube-api-access-qrjxr\") pod \"multus-additional-cni-plugins-h2lwh\" (UID: \"d604d7d6-843e-47f3-b837-31545356dbf3\") " pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.195210 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:38.195179    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957fcdbc1554bfacb8cd71b42c384138.slice/crio-49732af31831b7a5ffae6725ff3f54a43b3045210788ec324e99e80406a85219 WatchSource:0}: Error finding container 49732af31831b7a5ffae6725ff3f54a43b3045210788ec324e99e80406a85219: Status 404 returned error can't find the container with id 49732af31831b7a5ffae6725ff3f54a43b3045210788ec324e99e80406a85219
Jul 17 21:45:38.199256 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.199237    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:45:38.243590 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.243567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79e9c71a-027c-4087-a6e0-1ecf5753a7b5-tmp-dir\") pod \"node-resolver-sq8rg\" (UID: \"79e9c71a-027c-4087-a6e0-1ecf5753a7b5\") " pod="openshift-dns/node-resolver-sq8rg"
Jul 17 21:45:38.243665 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.243598    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-device-dir\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.243665 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.243614    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aeb396b3-2508-4c93-aeee-188c1b39fa9d-kubelet-config\") pod \"global-pull-secret-syncer-hf7sc\" (UID: \"aeb396b3-2508-4c93-aeee-188c1b39fa9d\") " pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:38.243665 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.243632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdv96\" (UniqueName: \"kubernetes.io/projected/79e9c71a-027c-4087-a6e0-1ecf5753a7b5-kube-api-access-rdv96\") pod \"node-resolver-sq8rg\" (UID: \"79e9c71a-027c-4087-a6e0-1ecf5753a7b5\") " pod="openshift-dns/node-resolver-sq8rg"
Jul 17 21:45:38.243788 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.243708    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aeb396b3-2508-4c93-aeee-188c1b39fa9d-kubelet-config\") pod \"global-pull-secret-syncer-hf7sc\" (UID: \"aeb396b3-2508-4c93-aeee-188c1b39fa9d\") " pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:38.243788 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.243723    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-device-dir\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.243788 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.243777    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjxgd\" (UniqueName: \"kubernetes.io/projected/b77bccab-4a6d-4e18-8206-9ad86e070317-kube-api-access-zjxgd\") pod \"node-ca-ls9f6\" (UID: \"b77bccab-4a6d-4e18-8206-9ad86e070317\") " pod="openshift-image-registry/node-ca-ls9f6"
Jul 17 21:45:38.243885 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.243805    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aeb396b3-2508-4c93-aeee-188c1b39fa9d-dbus\") pod \"global-pull-secret-syncer-hf7sc\" (UID: \"aeb396b3-2508-4c93-aeee-188c1b39fa9d\") " pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:38.243885 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.243871    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aeb396b3-2508-4c93-aeee-188c1b39fa9d-dbus\") pod \"global-pull-secret-syncer-hf7sc\" (UID: \"aeb396b3-2508-4c93-aeee-188c1b39fa9d\") " pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:38.243950 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.243898    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00ccd718-0d99-4948-84b4-6d0aaff1598b-konnectivity-ca\") pod \"konnectivity-agent-5xm2r\" (UID: \"00ccd718-0d99-4948-84b4-6d0aaff1598b\") " pod="kube-system/konnectivity-agent-5xm2r"
Jul 17 21:45:38.243988 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.243965    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-sys\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.244050 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00ccd718-0d99-4948-84b4-6d0aaff1598b-agent-certs\") pod \"konnectivity-agent-5xm2r\" (UID: \"00ccd718-0d99-4948-84b4-6d0aaff1598b\") " pod="kube-system/konnectivity-agent-5xm2r"
Jul 17 21:45:38.244050 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244009    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79e9c71a-027c-4087-a6e0-1ecf5753a7b5-tmp-dir\") pod \"node-resolver-sq8rg\" (UID: \"79e9c71a-027c-4087-a6e0-1ecf5753a7b5\") " pod="openshift-dns/node-resolver-sq8rg"
Jul 17 21:45:38.244165 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244103    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-sys\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.244165 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244110    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-lib-modules\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.244165 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244148    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-kubernetes\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.244305 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-host\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.244305 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244192    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx7wg\" (UniqueName: \"kubernetes.io/projected/1c3a69f3-bd78-488e-8f56-8f26361e5165-kube-api-access-zx7wg\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.244305 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244219    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-registration-dir\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.244305 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244237    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-kubernetes\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.244305 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244243    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.244545 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-lib-modules\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.244545 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b77bccab-4a6d-4e18-8206-9ad86e070317-serviceca\") pod \"node-ca-ls9f6\" (UID: \"b77bccab-4a6d-4e18-8206-9ad86e070317\") " pod="openshift-image-registry/node-ca-ls9f6"
Jul 17 21:45:38.244545 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244412    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-etc-selinux\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.244545 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-socket-dir\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.244545 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244457    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00ccd718-0d99-4948-84b4-6d0aaff1598b-konnectivity-ca\") pod \"konnectivity-agent-5xm2r\" (UID: \"00ccd718-0d99-4948-84b4-6d0aaff1598b\") " pod="kube-system/konnectivity-agent-5xm2r"
Jul 17 21:45:38.244545 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244470    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-registration-dir\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.244545 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244492    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-host\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.244545 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8q86\" (UniqueName: \"kubernetes.io/projected/14e20aba-c31d-4e1d-8a83-9ff790ede00a-kube-api-access-q8q86\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.244934 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244619    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-socket-dir\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.244934 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244688    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-sysctl-conf\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.244934 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244723    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/79e9c71a-027c-4087-a6e0-1ecf5753a7b5-hosts-file\") pod \"node-resolver-sq8rg\" (UID: \"79e9c71a-027c-4087-a6e0-1ecf5753a7b5\") " pod="openshift-dns/node-resolver-sq8rg"
Jul 17 21:45:38.244934 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244748    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.244934 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244796    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-systemd\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.244934 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244817    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/79e9c71a-027c-4087-a6e0-1ecf5753a7b5-hosts-file\") pod \"node-resolver-sq8rg\" (UID: \"79e9c71a-027c-4087-a6e0-1ecf5753a7b5\") " pod="openshift-dns/node-resolver-sq8rg"
Jul 17 21:45:38.244934 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-var-lib-kubelet\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.244934 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244862    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-var-lib-kubelet\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.244934 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244832    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b77bccab-4a6d-4e18-8206-9ad86e070317-serviceca\") pod \"node-ca-ls9f6\" (UID: \"b77bccab-4a6d-4e18-8206-9ad86e070317\") " pod="openshift-image-registry/node-ca-ls9f6"
Jul 17 21:45:38.244934 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244880    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-modprobe-d\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.244934 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244881    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.244934 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244891    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-sysctl-conf\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.244934 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244907    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-systemd\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.244934 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244912    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-tuned\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.245647 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244965    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-sys-fs\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.245647 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244984    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-modprobe-d\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.245647 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.244985    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret\") pod \"global-pull-secret-syncer-hf7sc\" (UID: \"aeb396b3-2508-4c93-aeee-188c1b39fa9d\") " pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:38.245647 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.245022    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-sysctl-d\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.245647 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.245030    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/14e20aba-c31d-4e1d-8a83-9ff790ede00a-sys-fs\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.245647 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.245046    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-run\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.245647 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.245058    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:45:38.245647 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.245106    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c3a69f3-bd78-488e-8f56-8f26361e5165-tmp\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.245647 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.245134    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret podName:aeb396b3-2508-4c93-aeee-188c1b39fa9d nodeName:}" failed. No retries permitted until 2026-07-17 21:45:38.745118495 +0000 UTC m=+2.134135138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret") pod "global-pull-secret-syncer-hf7sc" (UID: "aeb396b3-2508-4c93-aeee-188c1b39fa9d") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:45:38.245647 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.245147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-sysctl-d\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.245647 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.245154    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b77bccab-4a6d-4e18-8206-9ad86e070317-host\") pod \"node-ca-ls9f6\" (UID: \"b77bccab-4a6d-4e18-8206-9ad86e070317\") " pod="openshift-image-registry/node-ca-ls9f6"
Jul 17 21:45:38.245647 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.245187    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-sysconfig\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.245647 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.245193    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b77bccab-4a6d-4e18-8206-9ad86e070317-host\") pod \"node-ca-ls9f6\" (UID: \"b77bccab-4a6d-4e18-8206-9ad86e070317\") " pod="openshift-image-registry/node-ca-ls9f6"
Jul 17 21:45:38.245647 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.245133    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-run\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.245647 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.245270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-sysconfig\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.246232 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:38.245796    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1502cc83beae535f93cdbb53034f5e.slice/crio-15bc32d4112831995690d306fb4ca412412f3c0bbe375a16afc8c3b1db963596 WatchSource:0}: Error finding container 15bc32d4112831995690d306fb4ca412412f3c0bbe375a16afc8c3b1db963596: Status 404 returned error can't find the container with id 15bc32d4112831995690d306fb4ca412412f3c0bbe375a16afc8c3b1db963596
Jul 17 21:45:38.246657 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.246630    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00ccd718-0d99-4948-84b4-6d0aaff1598b-agent-certs\") pod \"konnectivity-agent-5xm2r\" (UID: \"00ccd718-0d99-4948-84b4-6d0aaff1598b\") " pod="kube-system/konnectivity-agent-5xm2r"
Jul 17 21:45:38.247377 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.247359    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c3a69f3-bd78-488e-8f56-8f26361e5165-etc-tuned\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.247471 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.247407    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c3a69f3-bd78-488e-8f56-8f26361e5165-tmp\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.251668 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.251648    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdv96\" (UniqueName: \"kubernetes.io/projected/79e9c71a-027c-4087-a6e0-1ecf5753a7b5-kube-api-access-rdv96\") pod \"node-resolver-sq8rg\" (UID: \"79e9c71a-027c-4087-a6e0-1ecf5753a7b5\") " pod="openshift-dns/node-resolver-sq8rg"
Jul 17 21:45:38.251973 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.251923    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjxgd\" (UniqueName: \"kubernetes.io/projected/b77bccab-4a6d-4e18-8206-9ad86e070317-kube-api-access-zjxgd\") pod \"node-ca-ls9f6\" (UID: \"b77bccab-4a6d-4e18-8206-9ad86e070317\") " pod="openshift-image-registry/node-ca-ls9f6"
Jul 17 21:45:38.253522 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.253495    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8q86\" (UniqueName: \"kubernetes.io/projected/14e20aba-c31d-4e1d-8a83-9ff790ede00a-kube-api-access-q8q86\") pod \"aws-ebs-csi-driver-node-xr9qm\" (UID: \"14e20aba-c31d-4e1d-8a83-9ff790ede00a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.253931 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.253914    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx7wg\" (UniqueName: \"kubernetes.io/projected/1c3a69f3-bd78-488e-8f56-8f26361e5165-kube-api-access-zx7wg\") pod \"tuned-6mrs7\" (UID: \"1c3a69f3-bd78-488e-8f56-8f26361e5165\") " pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.359676 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.359632    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qnscl"
Jul 17 21:45:38.370709 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:38.370666    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf619dbf4_d977_437e_9c8c_bd7eaf044562.slice/crio-8a829d1206db2fb678e6d47ffbe23ba5cc369190fa48de0db3eaf14d56cb9530 WatchSource:0}: Error finding container 8a829d1206db2fb678e6d47ffbe23ba5cc369190fa48de0db3eaf14d56cb9530: Status 404 returned error can't find the container with id 8a829d1206db2fb678e6d47ffbe23ba5cc369190fa48de0db3eaf14d56cb9530
Jul 17 21:45:38.387888 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.387862    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h2lwh"
Jul 17 21:45:38.397203 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:38.397174    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd604d7d6_843e_47f3_b837_31545356dbf3.slice/crio-492267fa771c19f8a594574e026176746940945a54df1b7aed10f98a8b6d1014 WatchSource:0}: Error finding container 492267fa771c19f8a594574e026176746940945a54df1b7aed10f98a8b6d1014: Status 404 returned error can't find the container with id 492267fa771c19f8a594574e026176746940945a54df1b7aed10f98a8b6d1014
Jul 17 21:45:38.403634 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.403613    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:45:38.412406 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:38.412377    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce379ff_d4b8_443c_8a1d_b522b7f9f5e6.slice/crio-7dc463d6b6ebd362431d2a52f852811689b136556cc839a3816afaf4c9e0aaff WatchSource:0}: Error finding container 7dc463d6b6ebd362431d2a52f852811689b136556cc839a3816afaf4c9e0aaff: Status 404 returned error can't find the container with id 7dc463d6b6ebd362431d2a52f852811689b136556cc839a3816afaf4c9e0aaff
Jul 17 21:45:38.413756 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.413722    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p2d44"
Jul 17 21:45:38.424157 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:38.424115    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05b7f74_b162_43d1_b600_53c5904268de.slice/crio-1b4da3a34942a5ec90c52f2b78add7d69b7d2e3baabe8b9246afff6795b10673 WatchSource:0}: Error finding container 1b4da3a34942a5ec90c52f2b78add7d69b7d2e3baabe8b9246afff6795b10673: Status 404 returned error can't find the container with id 1b4da3a34942a5ec90c52f2b78add7d69b7d2e3baabe8b9246afff6795b10673
Jul 17 21:45:38.436608 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.436584    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm"
Jul 17 21:45:38.445925 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:38.445890    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e20aba_c31d_4e1d_8a83_9ff790ede00a.slice/crio-16e4b9342aa56ef3b54ba81d56ce721e0c740ebc2e27e6309ce04a5ac3c8b30b WatchSource:0}: Error finding container 16e4b9342aa56ef3b54ba81d56ce721e0c740ebc2e27e6309ce04a5ac3c8b30b: Status 404 returned error can't find the container with id 16e4b9342aa56ef3b54ba81d56ce721e0c740ebc2e27e6309ce04a5ac3c8b30b
Jul 17 21:45:38.454857 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.454833    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5xm2r"
Jul 17 21:45:38.464490 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:38.464464    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ccd718_0d99_4948_84b4_6d0aaff1598b.slice/crio-b53a98f72b24704b16fc32faa0efcf58c9a8aba0a205ab0651a3cb0785715210 WatchSource:0}: Error finding container b53a98f72b24704b16fc32faa0efcf58c9a8aba0a205ab0651a3cb0785715210: Status 404 returned error can't find the container with id b53a98f72b24704b16fc32faa0efcf58c9a8aba0a205ab0651a3cb0785715210
Jul 17 21:45:38.476876 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.476848    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6mrs7"
Jul 17 21:45:38.485645 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:38.485614    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3a69f3_bd78_488e_8f56_8f26361e5165.slice/crio-9965c690a48ce43469f3ea10a2b2e417440e0a380239ab9046388b50b404308c WatchSource:0}: Error finding container 9965c690a48ce43469f3ea10a2b2e417440e0a380239ab9046388b50b404308c: Status 404 returned error can't find the container with id 9965c690a48ce43469f3ea10a2b2e417440e0a380239ab9046388b50b404308c
Jul 17 21:45:38.496744 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.496721    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sq8rg"
Jul 17 21:45:38.500773 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.500688    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ls9f6"
Jul 17 21:45:38.507756 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:38.507725    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e9c71a_027c_4087_a6e0_1ecf5753a7b5.slice/crio-ec74ee1b1fc33691be1fcf7806d38fe8bf91d2cf4336d904d04d37e2539d4887 WatchSource:0}: Error finding container ec74ee1b1fc33691be1fcf7806d38fe8bf91d2cf4336d904d04d37e2539d4887: Status 404 returned error can't find the container with id ec74ee1b1fc33691be1fcf7806d38fe8bf91d2cf4336d904d04d37e2539d4887
Jul 17 21:45:38.510362 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:45:38.510339    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77bccab_4a6d_4e18_8206_9ad86e070317.slice/crio-4d2c0fe3ba206752cd7e6c2ca0209f23f867ee649f367a8e50f9c9f07a19456f WatchSource:0}: Error finding container 4d2c0fe3ba206752cd7e6c2ca0209f23f867ee649f367a8e50f9c9f07a19456f: Status 404 returned error can't find the container with id 4d2c0fe3ba206752cd7e6c2ca0209f23f867ee649f367a8e50f9c9f07a19456f
Jul 17 21:45:38.646925 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.646883    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs\") pod \"network-metrics-daemon-cd95t\" (UID: \"ab648b99-a119-46ea-894c-2c83d2658261\") " pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:38.647134 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.647031    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:45:38.647134 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.647116    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs podName:ab648b99-a119-46ea-894c-2c83d2658261 nodeName:}" failed. No retries permitted until 2026-07-17 21:45:39.647101645 +0000 UTC m=+3.036118276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs") pod "network-metrics-daemon-cd95t" (UID: "ab648b99-a119-46ea-894c-2c83d2658261") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:45:38.747744 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.747708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x456\" (UniqueName: \"kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456\") pod \"network-check-target-mg8bv\" (UID: \"7e18d002-d4f9-49ff-b425-fc3c56585a80\") " pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:38.747923 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:38.747818    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret\") pod \"global-pull-secret-syncer-hf7sc\" (UID: \"aeb396b3-2508-4c93-aeee-188c1b39fa9d\") " pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:38.747923 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.747858    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:45:38.747923 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.747883    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:45:38.747923 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.747896    2565 projected.go:196] Error preparing data for projected volume kube-api-access-2x456 for pod openshift-network-diagnostics/network-check-target-mg8bv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:45:38.747923 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.747905    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:45:38.748107 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.747954    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456 podName:7e18d002-d4f9-49ff-b425-fc3c56585a80 nodeName:}" failed. No retries permitted until 2026-07-17 21:45:39.747934421 +0000 UTC m=+3.136951058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2x456" (UniqueName: "kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456") pod "network-check-target-mg8bv" (UID: "7e18d002-d4f9-49ff-b425-fc3c56585a80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:45:38.748107 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:38.747970    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret podName:aeb396b3-2508-4c93-aeee-188c1b39fa9d nodeName:}" failed. No retries permitted until 2026-07-17 21:45:39.747964076 +0000 UTC m=+3.136980706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret") pod "global-pull-secret-syncer-hf7sc" (UID: "aeb396b3-2508-4c93-aeee-188c1b39fa9d") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:45:39.035917 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.035880    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:45:39.080478 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.080257    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:40:38 +0000 UTC" deadline="2028-01-26 07:41:35.403167779 +0000 UTC"
Jul 17 21:45:39.080478 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.080325    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13377h55m56.322848048s"
Jul 17 21:45:39.178850 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.178814    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:39.179037 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:39.178972    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:45:39.187119 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.187057    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-11.ec2.internal" event={"ID":"1f1502cc83beae535f93cdbb53034f5e","Type":"ContainerStarted","Data":"15bc32d4112831995690d306fb4ca412412f3c0bbe375a16afc8c3b1db963596"}
Jul 17 21:45:39.195021 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.194986    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sq8rg" event={"ID":"79e9c71a-027c-4087-a6e0-1ecf5753a7b5","Type":"ContainerStarted","Data":"ec74ee1b1fc33691be1fcf7806d38fe8bf91d2cf4336d904d04d37e2539d4887"}
Jul 17 21:45:39.198257 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.198182    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5xm2r" event={"ID":"00ccd718-0d99-4948-84b4-6d0aaff1598b","Type":"ContainerStarted","Data":"b53a98f72b24704b16fc32faa0efcf58c9a8aba0a205ab0651a3cb0785715210"}
Jul 17 21:45:39.201390 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.201360    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj" event={"ID":"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6","Type":"ContainerStarted","Data":"7dc463d6b6ebd362431d2a52f852811689b136556cc839a3816afaf4c9e0aaff"}
Jul 17 21:45:39.210270 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.210241    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2lwh" event={"ID":"d604d7d6-843e-47f3-b837-31545356dbf3","Type":"ContainerStarted","Data":"492267fa771c19f8a594574e026176746940945a54df1b7aed10f98a8b6d1014"}
Jul 17 21:45:39.212249 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.212161    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal" event={"ID":"957fcdbc1554bfacb8cd71b42c384138","Type":"ContainerStarted","Data":"49732af31831b7a5ffae6725ff3f54a43b3045210788ec324e99e80406a85219"}
Jul 17 21:45:39.216061 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.216032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ls9f6" event={"ID":"b77bccab-4a6d-4e18-8206-9ad86e070317","Type":"ContainerStarted","Data":"4d2c0fe3ba206752cd7e6c2ca0209f23f867ee649f367a8e50f9c9f07a19456f"}
Jul 17 21:45:39.221143 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.221115    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6mrs7" event={"ID":"1c3a69f3-bd78-488e-8f56-8f26361e5165","Type":"ContainerStarted","Data":"9965c690a48ce43469f3ea10a2b2e417440e0a380239ab9046388b50b404308c"}
Jul 17 21:45:39.223609 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.223568    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm" event={"ID":"14e20aba-c31d-4e1d-8a83-9ff790ede00a","Type":"ContainerStarted","Data":"16e4b9342aa56ef3b54ba81d56ce721e0c740ebc2e27e6309ce04a5ac3c8b30b"}
Jul 17 21:45:39.227331 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.227301    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p2d44" event={"ID":"a05b7f74-b162-43d1-b600-53c5904268de","Type":"ContainerStarted","Data":"1b4da3a34942a5ec90c52f2b78add7d69b7d2e3baabe8b9246afff6795b10673"}
Jul 17 21:45:39.230726 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.230692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qnscl" event={"ID":"f619dbf4-d977-437e-9c8c-bd7eaf044562","Type":"ContainerStarted","Data":"8a829d1206db2fb678e6d47ffbe23ba5cc369190fa48de0db3eaf14d56cb9530"}
Jul 17 21:45:39.411118 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.410810    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:45:39.656188 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.656124    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs\") pod \"network-metrics-daemon-cd95t\" (UID: \"ab648b99-a119-46ea-894c-2c83d2658261\") " pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:39.656416 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:39.656279    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:45:39.656416 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:39.656347    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs podName:ab648b99-a119-46ea-894c-2c83d2658261 nodeName:}" failed. No retries permitted until 2026-07-17 21:45:41.656326318 +0000 UTC m=+5.045342954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs") pod "network-metrics-daemon-cd95t" (UID: "ab648b99-a119-46ea-894c-2c83d2658261") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:45:39.756586 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.756543    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret\") pod \"global-pull-secret-syncer-hf7sc\" (UID: \"aeb396b3-2508-4c93-aeee-188c1b39fa9d\") " pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:39.756793 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:39.756629    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x456\" (UniqueName: \"kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456\") pod \"network-check-target-mg8bv\" (UID: \"7e18d002-d4f9-49ff-b425-fc3c56585a80\") " pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:39.756869 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:39.756847    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:45:39.756927 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:39.756870    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:45:39.756927 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:39.756884    2565 projected.go:196] Error preparing data for projected volume kube-api-access-2x456 for pod openshift-network-diagnostics/network-check-target-mg8bv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:45:39.757039 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:39.756947    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456 podName:7e18d002-d4f9-49ff-b425-fc3c56585a80 nodeName:}" failed. No retries permitted until 2026-07-17 21:45:41.756927297 +0000 UTC m=+5.145943943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2x456" (UniqueName: "kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456") pod "network-check-target-mg8bv" (UID: "7e18d002-d4f9-49ff-b425-fc3c56585a80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:45:39.757432 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:39.757412    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:45:39.757502 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:39.757485    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret podName:aeb396b3-2508-4c93-aeee-188c1b39fa9d nodeName:}" failed. No retries permitted until 2026-07-17 21:45:41.757460393 +0000 UTC m=+5.146477027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret") pod "global-pull-secret-syncer-hf7sc" (UID: "aeb396b3-2508-4c93-aeee-188c1b39fa9d") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:45:40.081583 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:40.081471    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:40:38 +0000 UTC" deadline="2028-03-03 21:31:15.428842016 +0000 UTC"
Jul 17 21:45:40.081583 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:40.081518    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14279h45m35.347329647s"
Jul 17 21:45:40.176572 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:40.176347    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:40.176572 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:40.176488    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:45:40.176985 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:40.176961    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:40.177317 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:40.177071    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:45:41.002677 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:41.002367    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:45:41.178904 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:41.178379    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:41.178904 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:41.178530    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:45:41.679115 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:41.679060    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs\") pod \"network-metrics-daemon-cd95t\" (UID: \"ab648b99-a119-46ea-894c-2c83d2658261\") " pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:41.679348 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:41.679305    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:45:41.679417 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:41.679373    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs podName:ab648b99-a119-46ea-894c-2c83d2658261 nodeName:}" failed. No retries permitted until 2026-07-17 21:45:45.679352464 +0000 UTC m=+9.068369098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs") pod "network-metrics-daemon-cd95t" (UID: "ab648b99-a119-46ea-894c-2c83d2658261") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:45:41.779590 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:41.779545    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret\") pod \"global-pull-secret-syncer-hf7sc\" (UID: \"aeb396b3-2508-4c93-aeee-188c1b39fa9d\") " pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:41.779788 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:41.779622    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x456\" (UniqueName: \"kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456\") pod \"network-check-target-mg8bv\" (UID: \"7e18d002-d4f9-49ff-b425-fc3c56585a80\") " pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:41.779864 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:41.779805    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:45:41.779864 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:41.779828    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:45:41.779864 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:41.779851    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:45:41.779864 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:41.779864    2565 projected.go:196] Error preparing data for projected volume kube-api-access-2x456 for pod openshift-network-diagnostics/network-check-target-mg8bv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:45:41.780069 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:41.779885    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret podName:aeb396b3-2508-4c93-aeee-188c1b39fa9d nodeName:}" failed. No retries permitted until 2026-07-17 21:45:45.779863842 +0000 UTC m=+9.168880476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret") pod "global-pull-secret-syncer-hf7sc" (UID: "aeb396b3-2508-4c93-aeee-188c1b39fa9d") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:45:41.780069 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:41.779914    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456 podName:7e18d002-d4f9-49ff-b425-fc3c56585a80 nodeName:}" failed. No retries permitted until 2026-07-17 21:45:45.779898968 +0000 UTC m=+9.168915624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2x456" (UniqueName: "kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456") pod "network-check-target-mg8bv" (UID: "7e18d002-d4f9-49ff-b425-fc3c56585a80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:45:42.176979 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:42.176934    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:42.177178 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:42.177086    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:45:42.177178 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:42.177175    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:42.177313 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:42.177252    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:45:43.178325 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:43.178279    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:43.178854 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:43.178484    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:45:44.176637 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:44.176602    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:44.176831 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:44.176703    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:45:44.176831 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:44.176758    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:44.176831 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:44.176801    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:45:45.176615 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:45.176586    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:45.177011 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:45.176720    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:45:45.713723 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:45.713671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs\") pod \"network-metrics-daemon-cd95t\" (UID: \"ab648b99-a119-46ea-894c-2c83d2658261\") " pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:45.713905 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:45.713848    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:45:45.713968 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:45.713927    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs podName:ab648b99-a119-46ea-894c-2c83d2658261 nodeName:}" failed. No retries permitted until 2026-07-17 21:45:53.713905156 +0000 UTC m=+17.102921791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs") pod "network-metrics-daemon-cd95t" (UID: "ab648b99-a119-46ea-894c-2c83d2658261") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:45:45.814825 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:45.814776    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x456\" (UniqueName: \"kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456\") pod \"network-check-target-mg8bv\" (UID: \"7e18d002-d4f9-49ff-b425-fc3c56585a80\") " pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:45.815006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:45.814869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret\") pod \"global-pull-secret-syncer-hf7sc\" (UID: \"aeb396b3-2508-4c93-aeee-188c1b39fa9d\") " pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:45.815006 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:45.814990    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:45:45.815146 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:45.815052    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret podName:aeb396b3-2508-4c93-aeee-188c1b39fa9d nodeName:}" failed. No retries permitted until 2026-07-17 21:45:53.815031789 +0000 UTC m=+17.204048436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret") pod "global-pull-secret-syncer-hf7sc" (UID: "aeb396b3-2508-4c93-aeee-188c1b39fa9d") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:45:45.815212 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:45.815161    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:45:45.815212 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:45.815175    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:45:45.815212 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:45.815188    2565 projected.go:196] Error preparing data for projected volume kube-api-access-2x456 for pod openshift-network-diagnostics/network-check-target-mg8bv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:45:45.815357 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:45.815228    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456 podName:7e18d002-d4f9-49ff-b425-fc3c56585a80 nodeName:}" failed. No retries permitted until 2026-07-17 21:45:53.81521483 +0000 UTC m=+17.204231464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2x456" (UniqueName: "kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456") pod "network-check-target-mg8bv" (UID: "7e18d002-d4f9-49ff-b425-fc3c56585a80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:45:46.177023 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:46.176989    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:46.177519 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:46.177135    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:45:46.177519 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:46.177201    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:46.177519 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:46.177270    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:45:47.177622 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:47.177581    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:47.178030 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:47.177720    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:45:48.177049 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:48.177004    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:48.177255 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:48.177159    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:45:48.177586 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:48.177566    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:48.177740 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:48.177719    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:45:49.176548 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:49.176510    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:49.176742 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:49.176660    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:45:50.176893 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:50.176453    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:50.176893 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:50.176578    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:45:50.176893 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:50.176645    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:50.176893 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:50.176760    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:45:51.176088 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:51.176039    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:51.176273 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:51.176169    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:45:52.176338 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:52.176300    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:52.176829 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:52.176348    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:52.176829 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:52.176431    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:45:52.176829 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:52.176558    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:45:53.176677 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:53.176647    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:53.177152 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:53.176781    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:45:53.775571 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:53.775528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs\") pod \"network-metrics-daemon-cd95t\" (UID: \"ab648b99-a119-46ea-894c-2c83d2658261\") " pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:53.775775 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:53.775679    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:45:53.775775 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:53.775757    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs podName:ab648b99-a119-46ea-894c-2c83d2658261 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:09.775739863 +0000 UTC m=+33.164756492 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs") pod "network-metrics-daemon-cd95t" (UID: "ab648b99-a119-46ea-894c-2c83d2658261") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:45:53.876415 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:53.876378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret\") pod \"global-pull-secret-syncer-hf7sc\" (UID: \"aeb396b3-2508-4c93-aeee-188c1b39fa9d\") " pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:53.876612 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:53.876443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x456\" (UniqueName: \"kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456\") pod \"network-check-target-mg8bv\" (UID: \"7e18d002-d4f9-49ff-b425-fc3c56585a80\") " pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:53.876612 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:53.876534    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:45:53.876612 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:53.876580    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:45:53.876612 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:53.876598    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:45:53.876612 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:53.876607    2565 projected.go:196] Error preparing data for projected volume kube-api-access-2x456 for pod openshift-network-diagnostics/network-check-target-mg8bv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:45:53.876863 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:53.876615    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret podName:aeb396b3-2508-4c93-aeee-188c1b39fa9d nodeName:}" failed. No retries permitted until 2026-07-17 21:46:09.876595757 +0000 UTC m=+33.265612387 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret") pod "global-pull-secret-syncer-hf7sc" (UID: "aeb396b3-2508-4c93-aeee-188c1b39fa9d") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:45:53.876863 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:53.876644    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456 podName:7e18d002-d4f9-49ff-b425-fc3c56585a80 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:09.876635076 +0000 UTC m=+33.265651705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2x456" (UniqueName: "kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456") pod "network-check-target-mg8bv" (UID: "7e18d002-d4f9-49ff-b425-fc3c56585a80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:45:54.176480 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:54.176388    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:54.176480 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:54.176436    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:54.176703 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:54.176530    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:45:54.176703 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:54.176647    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:45:55.176136 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:55.176089    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:55.176356 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:55.176239    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:45:56.176288 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:56.176249    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:56.176759 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:56.176251    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:56.176759 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:56.176410    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:45:56.176759 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:56.176498    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:45:57.177622 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:57.177590    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:57.178054 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:57.177700    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:45:58.176303 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:58.176271    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:45:58.176493 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:58.176271    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:45:58.176493 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:58.176410    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:45:58.176493 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:58.176474    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:45:59.176798 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:45:59.176760    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:45:59.177215 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:45:59.176911    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:46:00.176968 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.176802    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:46:00.177497 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.176802    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:46:00.177497 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:00.177044    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:46:00.177497 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:00.177130    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:46:00.274469 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.274433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ls9f6" event={"ID":"b77bccab-4a6d-4e18-8206-9ad86e070317","Type":"ContainerStarted","Data":"d80bc08bb20e5f63a44d0e52397f3240281bc3bf6d4bfdfd05dbb4635d082289"}
Jul 17 21:46:00.276244 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.276162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6mrs7" event={"ID":"1c3a69f3-bd78-488e-8f56-8f26361e5165","Type":"ContainerStarted","Data":"da9ef996656debff2f1ad36cb3cbe87c0e3d026f3ee8936f0f16a86494eab23a"}
Jul 17 21:46:00.279173 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.278393    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm" event={"ID":"14e20aba-c31d-4e1d-8a83-9ff790ede00a","Type":"ContainerStarted","Data":"8bfd079a91207854cf3ef7197cfb18d6287c9f9e9bc340ee123a9b835a138e49"}
Jul 17 21:46:00.282316 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.282288    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p2d44" event={"ID":"a05b7f74-b162-43d1-b600-53c5904268de","Type":"ContainerStarted","Data":"9857ec139ad4db1b9dbdb44f16215679cf7ae6b59b6e43d3e21b35bc268e699e"}
Jul 17 21:46:00.283770 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.283744    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-11.ec2.internal" event={"ID":"1f1502cc83beae535f93cdbb53034f5e","Type":"ContainerStarted","Data":"1161586b58d846ae4e64b655e30f8b41b3b8d53f28d068998f082dad1c2f3eb0"}
Jul 17 21:46:00.285287 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.285264    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sq8rg" event={"ID":"79e9c71a-027c-4087-a6e0-1ecf5753a7b5","Type":"ContainerStarted","Data":"19fab13a8bde7d2d86013508726d8600da4cf0db3d035d17dbb9ed5eeefba1c6"}
Jul 17 21:46:00.286736 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.286703    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5xm2r" event={"ID":"00ccd718-0d99-4948-84b4-6d0aaff1598b","Type":"ContainerStarted","Data":"ff76e1523ac4cd683931e948c0c7dac013aea0bd89a16c15b2dd7adecb9efb40"}
Jul 17 21:46:00.287878 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.287831    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ls9f6" podStartSLOduration=2.395160708 podStartE2EDuration="23.287807046s" totalImagesPullingTime="20.892646338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:00.287651021 +0000 UTC m=+23.676667694" watchObservedRunningTime="2026-07-17 21:46:00.287807046 +0000 UTC m=+23.676823699"
Jul 17 21:46:00.289741 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.289722    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 21:46:00.290105 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.290080    2565 generic.go:356] "Generic (PLEG): container finished" podID="4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6" containerID="6425bbcc5b1a8632e57f3b6aac6e0767650a35fff6cde96cfc508e2eb277f8d4" exitCode=1
Jul 17 21:46:00.290170 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.290092    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj" event={"ID":"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6","Type":"ContainerStarted","Data":"424cffd5ff979080d73ff9576c278c9ad1b6865b116a5f13c1a434b6da734c5d"}
Jul 17 21:46:00.290230 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.290199    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj" event={"ID":"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6","Type":"ContainerStarted","Data":"4ca38c3e33808305c9a216b43fbe17c621d029f93ec566f6c3cafde1b48950f9"}
Jul 17 21:46:00.290433 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.290250    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj" event={"ID":"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6","Type":"ContainerStarted","Data":"044bb704c4fdb0aefe509c8ae15c2fb87363491f0ef1f04ba994d1a2cac99f00"}
Jul 17 21:46:00.290433 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.290319    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj" event={"ID":"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6","Type":"ContainerStarted","Data":"c522f4e530a272b5c265b63461486f74405e579fed189295ea579a42850464c6"}
Jul 17 21:46:00.290433 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.290356    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj" event={"ID":"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6","Type":"ContainerDied","Data":"6425bbcc5b1a8632e57f3b6aac6e0767650a35fff6cde96cfc508e2eb277f8d4"}
Jul 17 21:46:00.290433 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.290397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj" event={"ID":"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6","Type":"ContainerStarted","Data":"74df1e8bf5abf2243c232bc1e50df0b778af11456dcb7e309350e92e1caacdbd"}
Jul 17 21:46:00.291645 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.291623    2565 generic.go:356] "Generic (PLEG): container finished" podID="d604d7d6-843e-47f3-b837-31545356dbf3" containerID="20426896bb5e140449df90b05f10c72d431e8a9ca5e745c8a2406beac4704520" exitCode=0
Jul 17 21:46:00.291736 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.291711    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2lwh" event={"ID":"d604d7d6-843e-47f3-b837-31545356dbf3","Type":"ContainerDied","Data":"20426896bb5e140449df90b05f10c72d431e8a9ca5e745c8a2406beac4704520"}
Jul 17 21:46:00.293452 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.293434    2565 generic.go:356] "Generic (PLEG): container finished" podID="957fcdbc1554bfacb8cd71b42c384138" containerID="b46ef200c5a8b143607a097c75214532039a3f30a6738bb092c788a940c307ae" exitCode=0
Jul 17 21:46:00.293550 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.293461    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal" event={"ID":"957fcdbc1554bfacb8cd71b42c384138","Type":"ContainerDied","Data":"b46ef200c5a8b143607a097c75214532039a3f30a6738bb092c788a940c307ae"}
Jul 17 21:46:00.313774 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.313732    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-11.ec2.internal" podStartSLOduration=22.31371523 podStartE2EDuration="22.31371523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:00.30029247 +0000 UTC m=+23.689309135" watchObservedRunningTime="2026-07-17 21:46:00.31371523 +0000 UTC m=+23.702731884"
Jul 17 21:46:00.335535 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.335497    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5xm2r" podStartSLOduration=2.39380086 podStartE2EDuration="23.335481175s" totalImagesPullingTime="20.941680315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:00.313708598 +0000 UTC m=+23.702725245" watchObservedRunningTime="2026-07-17 21:46:00.335481175 +0000 UTC m=+23.724497874"
Jul 17 21:46:00.335843 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.335822    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6mrs7" podStartSLOduration=2.404155392 podStartE2EDuration="23.335817847s" totalImagesPullingTime="20.931662455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:00.335345269 +0000 UTC m=+23.724361941" watchObservedRunningTime="2026-07-17 21:46:00.335817847 +0000 UTC m=+23.724834497"
Jul 17 21:46:00.351923 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.351868    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p2d44" podStartSLOduration=2.306136305 podStartE2EDuration="23.35185301s" totalImagesPullingTime="21.045716705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:00.351402525 +0000 UTC m=+23.740419177" watchObservedRunningTime="2026-07-17 21:46:00.35185301 +0000 UTC m=+23.740869661"
Jul 17 21:46:00.365852 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:00.365790    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sq8rg" podStartSLOduration=2.470591369 podStartE2EDuration="23.365769078s" totalImagesPullingTime="20.895177709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:00.365527958 +0000 UTC m=+23.754544609" watchObservedRunningTime="2026-07-17 21:46:00.365769078 +0000 UTC m=+23.754785733"
Jul 17 21:46:01.176648 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:01.176407    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:46:01.176828 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:01.176695    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:46:01.297234 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:01.297196    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qnscl" event={"ID":"f619dbf4-d977-437e-9c8c-bd7eaf044562","Type":"ContainerStarted","Data":"c180272f875cbb3d997bfb2809b56faca741e17db8aaa0f84ae6ca4b0169022b"}
Jul 17 21:46:01.299036 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:01.299007    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal" event={"ID":"957fcdbc1554bfacb8cd71b42c384138","Type":"ContainerStarted","Data":"bba1e859c925e8e54d2375b6bce44639e730a65b61247307a6356ce8a5c90875"}
Jul 17 21:46:01.312880 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:01.312822    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qnscl" podStartSLOduration=3.277333683 podStartE2EDuration="24.312797827s" totalImagesPullingTime="21.035464144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:01.312029256 +0000 UTC m=+24.701045910" watchObservedRunningTime="2026-07-17 21:46:01.312797827 +0000 UTC m=+24.701814480"
Jul 17 21:46:01.328981 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:01.328907    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-11.ec2.internal" podStartSLOduration=23.328884953 podStartE2EDuration="23.328884953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:01.327963118 +0000 UTC m=+24.716979770" watchObservedRunningTime="2026-07-17 21:46:01.328884953 +0000 UTC m=+24.717901605"
Jul 17 21:46:02.177117 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:02.177091    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:46:02.177263 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:02.177128    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:46:02.177263 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:02.177212    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:46:02.177365 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:02.177344    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:46:02.186462 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:02.186436    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 21:46:02.304152 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:02.304044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm" event={"ID":"14e20aba-c31d-4e1d-8a83-9ff790ede00a","Type":"ContainerStarted","Data":"49a46e02c7de3b9d93b6493e11270431b3fe84146bb20d205a4090e5404c8794"}
Jul 17 21:46:03.111251 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:03.111128    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T21:46:02.186457961Z","UUID":"7f9886e6-ac40-427b-b034-0b1bd558aa6e","Handler":null,"Name":"","Endpoint":""}
Jul 17 21:46:03.112909 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:03.112889    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 21:46:03.113036 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:03.112928    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 21:46:03.177200 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:03.177163    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:46:03.177375 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:03.177311    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:46:03.310246 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:03.310217    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 21:46:03.310675 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:03.310629    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj" event={"ID":"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6","Type":"ContainerStarted","Data":"fb4031e82a2af6a19408eed3989393d47bf8bd4879463e2c81100a688fbf3a7a"}
Jul 17 21:46:04.176240 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:04.176204    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:46:04.176443 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:04.176259    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:46:04.176443 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:04.176337    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:46:04.176530 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:04.176439    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:46:05.176713 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:05.176409    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:46:05.177144 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:05.176763    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:46:05.197398 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:05.197372    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5xm2r"
Jul 17 21:46:05.197916 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:05.197898    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5xm2r"
Jul 17 21:46:05.319439 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:05.319414    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 21:46:05.319915 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:05.319891    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj" event={"ID":"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6","Type":"ContainerStarted","Data":"a3cafbdbf837fb8a250f851120a31bc928f1a0583229e2544ed04ab732994b08"}
Jul 17 21:46:05.320255 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:05.320226    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:46:05.320446 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:05.320367    2565 scope.go:117] "RemoveContainer" containerID="6425bbcc5b1a8632e57f3b6aac6e0767650a35fff6cde96cfc508e2eb277f8d4"
Jul 17 21:46:05.322197 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:05.322175    2565 generic.go:356] "Generic (PLEG): container finished" podID="d604d7d6-843e-47f3-b837-31545356dbf3" containerID="7bfa0123bbfebdb9c06a7aec5b9a3134ab508033fb330563f604389edcc8e2b8" exitCode=0
Jul 17 21:46:05.322305 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:05.322261    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2lwh" event={"ID":"d604d7d6-843e-47f3-b837-31545356dbf3","Type":"ContainerDied","Data":"7bfa0123bbfebdb9c06a7aec5b9a3134ab508033fb330563f604389edcc8e2b8"}
Jul 17 21:46:05.322365 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:05.322349    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5xm2r"
Jul 17 21:46:05.322968 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:05.322865    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5xm2r"
Jul 17 21:46:05.336631 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:05.336611    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:46:06.176899 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:06.176822    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:46:06.177571 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:06.176826    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:46:06.177571 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:06.176949    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:46:06.177571 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:06.176991    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:46:06.329275 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:06.329246    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 21:46:06.329777 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:06.329746    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj" event={"ID":"4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6","Type":"ContainerStarted","Data":"d5e9aaa76bd5667e91f998881fa6f8b83be82e929f6fed062559c72794339413"}
Jul 17 21:46:06.330038 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:06.330019    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:46:06.330161 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:06.330115    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:46:06.332842 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:06.332812    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm" event={"ID":"14e20aba-c31d-4e1d-8a83-9ff790ede00a","Type":"ContainerStarted","Data":"c6752dd0d00b0f4b412e737fb9ae9f22e75168575e5b2bba5baaadcb3b7e81a2"}
Jul 17 21:46:06.348446 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:06.348416    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:46:06.359629 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:06.359586    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj" podStartSLOduration=8.295383951 podStartE2EDuration="29.359569933s" totalImagesPullingTime="21.064185982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:06.35917498 +0000 UTC m=+29.748191650" watchObservedRunningTime="2026-07-17 21:46:06.359569933 +0000 UTC m=+29.748586584"
Jul 17 21:46:06.404840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:06.404648    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xr9qm" podStartSLOduration=2.682829854 podStartE2EDuration="29.404631532s" totalImagesPullingTime="26.721801678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:45:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:06.37738016 +0000 UTC m=+29.766396822" watchObservedRunningTime="2026-07-17 21:46:06.404631532 +0000 UTC m=+29.793648183"
Jul 17 21:46:06.583622 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:06.583583    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hf7sc"]
Jul 17 21:46:06.583779 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:06.583691    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:46:06.583822 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:06.583780    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:46:06.586820 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:06.586793    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mg8bv"]
Jul 17 21:46:06.586981 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:06.586882    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:46:06.586981 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:06.586974    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:46:06.587474 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:06.587454    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cd95t"]
Jul 17 21:46:06.587553 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:06.587539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:46:06.587632 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:06.587609    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:46:07.336698 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:07.336650    2565 generic.go:356] "Generic (PLEG): container finished" podID="d604d7d6-843e-47f3-b837-31545356dbf3" containerID="f7e0c98c43ff95420a87ac7663e97274f8b7abe99c8bc15710a2bf24fddf2e78" exitCode=0
Jul 17 21:46:07.337191 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:07.336759    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2lwh" event={"ID":"d604d7d6-843e-47f3-b837-31545356dbf3","Type":"ContainerDied","Data":"f7e0c98c43ff95420a87ac7663e97274f8b7abe99c8bc15710a2bf24fddf2e78"}
Jul 17 21:46:08.176594 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:08.176549    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:46:08.176800 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:08.176601    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:46:08.176800 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:08.176601    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:46:08.176800 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:08.176682    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:46:08.176800 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:08.176764    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:46:08.176972 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:08.176891    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:46:09.344108 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:09.344044    2565 generic.go:356] "Generic (PLEG): container finished" podID="d604d7d6-843e-47f3-b837-31545356dbf3" containerID="d8a1db6bffe708b387e2d4b95e07fcf08bf5c5ee5a93e104081fd6b1d6c5142d" exitCode=0
Jul 17 21:46:09.344108 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:09.344097    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2lwh" event={"ID":"d604d7d6-843e-47f3-b837-31545356dbf3","Type":"ContainerDied","Data":"d8a1db6bffe708b387e2d4b95e07fcf08bf5c5ee5a93e104081fd6b1d6c5142d"}
Jul 17 21:46:09.789283 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:09.789253    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs\") pod \"network-metrics-daemon-cd95t\" (UID: \"ab648b99-a119-46ea-894c-2c83d2658261\") " pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:46:09.789455 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:09.789372    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:46:09.789455 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:09.789424    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs podName:ab648b99-a119-46ea-894c-2c83d2658261 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:41.78941013 +0000 UTC m=+65.178426763 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs") pod "network-metrics-daemon-cd95t" (UID: "ab648b99-a119-46ea-894c-2c83d2658261") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:46:09.890640 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:09.890601    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret\") pod \"global-pull-secret-syncer-hf7sc\" (UID: \"aeb396b3-2508-4c93-aeee-188c1b39fa9d\") " pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:46:09.890816 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:09.890670    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x456\" (UniqueName: \"kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456\") pod \"network-check-target-mg8bv\" (UID: \"7e18d002-d4f9-49ff-b425-fc3c56585a80\") " pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:46:09.890816 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:09.890791    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:46:09.890969 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:09.890835    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:46:09.890969 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:09.890851    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:46:09.890969 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:09.890864    2565 projected.go:196] Error preparing data for projected volume kube-api-access-2x456 for pod openshift-network-diagnostics/network-check-target-mg8bv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:46:09.890969 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:09.890866    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret podName:aeb396b3-2508-4c93-aeee-188c1b39fa9d nodeName:}" failed. No retries permitted until 2026-07-17 21:46:41.890844135 +0000 UTC m=+65.279860785 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret") pod "global-pull-secret-syncer-hf7sc" (UID: "aeb396b3-2508-4c93-aeee-188c1b39fa9d") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:46:09.890969 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:09.890910    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456 podName:7e18d002-d4f9-49ff-b425-fc3c56585a80 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:41.890894239 +0000 UTC m=+65.279910885 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2x456" (UniqueName: "kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456") pod "network-check-target-mg8bv" (UID: "7e18d002-d4f9-49ff-b425-fc3c56585a80") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:46:10.177061 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:10.176982    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:46:10.177236 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:10.176982    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:46:10.177236 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:10.177136    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:46:10.177236 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:10.176982    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:46:10.177236 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:10.177185    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:46:10.177452 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:10.177253    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:46:10.671515 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:10.671470    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:10.761369 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:10.761340    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:11.328140 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:11.328093    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:12.176696 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:12.176658    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:46:12.177153 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:12.176714    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:46:12.177153 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:12.176714    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:46:12.177153 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:12.176802    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hf7sc" podUID="aeb396b3-2508-4c93-aeee-188c1b39fa9d"
Jul 17 21:46:12.177153 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:12.176895    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mg8bv" podUID="7e18d002-d4f9-49ff-b425-fc3c56585a80"
Jul 17 21:46:12.177153 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:12.176973    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cd95t" podUID="ab648b99-a119-46ea-894c-2c83d2658261"
Jul 17 21:46:12.413428 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:12.413391    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:12.469348 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:12.469260    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:13.411336 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.411303    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-11.ec2.internal" event="NodeReady"
Jul 17 21:46:13.411777 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.411505    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 21:46:13.469309 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.469275    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54b875dd99-cs4hm"]
Jul 17 21:46:13.482396 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.482372    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8hzbf"]
Jul 17 21:46:13.482554 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.482499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.486016 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.485132    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 21:46:13.486016 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.485760    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6j6px\""
Jul 17 21:46:13.486016 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.486018    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 21:46:13.486382 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.486323    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 21:46:13.491751 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.491729    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 21:46:13.512848 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.512815    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-q5vtj"]
Jul 17 21:46:13.512968 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.512940    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:13.515536 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.515512    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 21:46:13.515536 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.515533    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q82nm\""
Jul 17 21:46:13.515695 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.515534    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 21:46:13.534344 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.534322    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54b875dd99-cs4hm"]
Jul 17 21:46:13.534344 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.534351    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q5vtj"]
Jul 17 21:46:13.534539 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.534365    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8hzbf"]
Jul 17 21:46:13.534539 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.534455    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q5vtj"
Jul 17 21:46:13.537184 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.537162    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 21:46:13.537338 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.537198    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 21:46:13.537338 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.537221    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l5qwj\""
Jul 17 21:46:13.537338 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.537167    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 21:46:13.622099 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.622053    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18e6a280-24ab-486f-9870-83b1c68cc38b-ca-trust-extracted\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.622099 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.622103    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18e6a280-24ab-486f-9870-83b1c68cc38b-installation-pull-secrets\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.622342 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.622179    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5stkm\" (UniqueName: \"kubernetes.io/projected/13be2284-1e6e-414b-bd17-01caea2dc51c-kube-api-access-5stkm\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:13.622342 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.622222    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-certificates\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.622342 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.622253    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mk4j\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-kube-api-access-9mk4j\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.622342 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.622311    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18e6a280-24ab-486f-9870-83b1c68cc38b-trusted-ca\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.622342 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.622341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:13.622556 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.622358    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13be2284-1e6e-414b-bd17-01caea2dc51c-tmp-dir\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:13.622556 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.622388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.622556 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.622415    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-bound-sa-token\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.622556 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.622435    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c961b41-034e-40be-a3f6-292e81593596-cert\") pod \"ingress-canary-q5vtj\" (UID: \"2c961b41-034e-40be-a3f6-292e81593596\") " pod="openshift-ingress-canary/ingress-canary-q5vtj"
Jul 17 21:46:13.622556 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.622473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18e6a280-24ab-486f-9870-83b1c68cc38b-image-registry-private-configuration\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.622556 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.622512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13be2284-1e6e-414b-bd17-01caea2dc51c-config-volume\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:13.622556 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.622545    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-467bp\" (UniqueName: \"kubernetes.io/projected/2c961b41-034e-40be-a3f6-292e81593596-kube-api-access-467bp\") pod \"ingress-canary-q5vtj\" (UID: \"2c961b41-034e-40be-a3f6-292e81593596\") " pod="openshift-ingress-canary/ingress-canary-q5vtj"
Jul 17 21:46:13.726125 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.723374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18e6a280-24ab-486f-9870-83b1c68cc38b-ca-trust-extracted\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.726125 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.723434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18e6a280-24ab-486f-9870-83b1c68cc38b-installation-pull-secrets\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.726125 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.723497    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5stkm\" (UniqueName: \"kubernetes.io/projected/13be2284-1e6e-414b-bd17-01caea2dc51c-kube-api-access-5stkm\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:13.726125 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.723535    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-certificates\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.726125 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.723567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mk4j\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-kube-api-access-9mk4j\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.726125 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.723610    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18e6a280-24ab-486f-9870-83b1c68cc38b-trusted-ca\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.726125 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.723634    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:13.726125 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.723664    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13be2284-1e6e-414b-bd17-01caea2dc51c-tmp-dir\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:13.726125 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.723716    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.726125 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.723747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-bound-sa-token\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.726125 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.723785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c961b41-034e-40be-a3f6-292e81593596-cert\") pod \"ingress-canary-q5vtj\" (UID: \"2c961b41-034e-40be-a3f6-292e81593596\") " pod="openshift-ingress-canary/ingress-canary-q5vtj"
Jul 17 21:46:13.726125 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.723817    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18e6a280-24ab-486f-9870-83b1c68cc38b-image-registry-private-configuration\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.726125 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.723854    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13be2284-1e6e-414b-bd17-01caea2dc51c-config-volume\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:13.726125 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.723884    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-467bp\" (UniqueName: \"kubernetes.io/projected/2c961b41-034e-40be-a3f6-292e81593596-kube-api-access-467bp\") pod \"ingress-canary-q5vtj\" (UID: \"2c961b41-034e-40be-a3f6-292e81593596\") " pod="openshift-ingress-canary/ingress-canary-q5vtj"
Jul 17 21:46:13.726125 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.724732    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18e6a280-24ab-486f-9870-83b1c68cc38b-ca-trust-extracted\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.726946 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.726214    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13be2284-1e6e-414b-bd17-01caea2dc51c-tmp-dir\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:13.726946 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:13.726615    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:46:13.726946 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:13.726631    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b875dd99-cs4hm: secret "image-registry-tls" not found
Jul 17 21:46:13.726946 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:13.726678    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls podName:18e6a280-24ab-486f-9870-83b1c68cc38b nodeName:}" failed. No retries permitted until 2026-07-17 21:46:14.22666372 +0000 UTC m=+37.615680349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls") pod "image-registry-54b875dd99-cs4hm" (UID: "18e6a280-24ab-486f-9870-83b1c68cc38b") : secret "image-registry-tls" not found
Jul 17 21:46:13.726946 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.726720    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13be2284-1e6e-414b-bd17-01caea2dc51c-config-volume\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:13.726946 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:13.726940    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:46:13.727264 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:13.727006    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls podName:13be2284-1e6e-414b-bd17-01caea2dc51c nodeName:}" failed. No retries permitted until 2026-07-17 21:46:14.226985467 +0000 UTC m=+37.616002113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls") pod "dns-default-8hzbf" (UID: "13be2284-1e6e-414b-bd17-01caea2dc51c") : secret "dns-default-metrics-tls" not found
Jul 17 21:46:13.727264 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.727007    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-certificates\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.727264 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.727205    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18e6a280-24ab-486f-9870-83b1c68cc38b-trusted-ca\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.730269 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.730244    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18e6a280-24ab-486f-9870-83b1c68cc38b-image-registry-private-configuration\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.730413 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.730287    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18e6a280-24ab-486f-9870-83b1c68cc38b-installation-pull-secrets\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.730413 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.730296    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c961b41-034e-40be-a3f6-292e81593596-cert\") pod \"ingress-canary-q5vtj\" (UID: \"2c961b41-034e-40be-a3f6-292e81593596\") " pod="openshift-ingress-canary/ingress-canary-q5vtj"
Jul 17 21:46:13.732164 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.732141    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-467bp\" (UniqueName: \"kubernetes.io/projected/2c961b41-034e-40be-a3f6-292e81593596-kube-api-access-467bp\") pod \"ingress-canary-q5vtj\" (UID: \"2c961b41-034e-40be-a3f6-292e81593596\") " pod="openshift-ingress-canary/ingress-canary-q5vtj"
Jul 17 21:46:13.737328 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.737306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5stkm\" (UniqueName: \"kubernetes.io/projected/13be2284-1e6e-414b-bd17-01caea2dc51c-kube-api-access-5stkm\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:13.737443 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.737338    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-bound-sa-token\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.737496 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.737477    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mk4j\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-kube-api-access-9mk4j\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:13.842802 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:13.842764    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q5vtj"
Jul 17 21:46:14.053162 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:14.052901    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q5vtj"]
Jul 17 21:46:14.055456 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:46:14.055424    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c961b41_034e_40be_a3f6_292e81593596.slice/crio-896c830a67ccf5d3ffedd44ebfe61cef7d7310f20b2b89c2f0a45f4727e344f8 WatchSource:0}: Error finding container 896c830a67ccf5d3ffedd44ebfe61cef7d7310f20b2b89c2f0a45f4727e344f8: Status 404 returned error can't find the container with id 896c830a67ccf5d3ffedd44ebfe61cef7d7310f20b2b89c2f0a45f4727e344f8
Jul 17 21:46:14.176730 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:14.176698    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:46:14.176868 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:14.176714    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:46:14.176906 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:14.176873    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:46:14.179445 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:14.179421    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 21:46:14.179574 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:14.179457    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 21:46:14.179574 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:14.179478    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 21:46:14.179761 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:14.179740    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mbv8k\""
Jul 17 21:46:14.179856 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:14.179793    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 21:46:14.179856 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:14.179811    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-28dc9\""
Jul 17 21:46:14.227859 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:14.227826    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:14.227975 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:14.227866    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:14.228030 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:14.227976    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:46:14.228030 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:14.227981    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:46:14.228030 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:14.227998    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b875dd99-cs4hm: secret "image-registry-tls" not found
Jul 17 21:46:14.228147 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:14.228033    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls podName:13be2284-1e6e-414b-bd17-01caea2dc51c nodeName:}" failed. No retries permitted until 2026-07-17 21:46:15.228018159 +0000 UTC m=+38.617034791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls") pod "dns-default-8hzbf" (UID: "13be2284-1e6e-414b-bd17-01caea2dc51c") : secret "dns-default-metrics-tls" not found
Jul 17 21:46:14.228147 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:14.228046    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls podName:18e6a280-24ab-486f-9870-83b1c68cc38b nodeName:}" failed. No retries permitted until 2026-07-17 21:46:15.228040615 +0000 UTC m=+38.617057244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls") pod "image-registry-54b875dd99-cs4hm" (UID: "18e6a280-24ab-486f-9870-83b1c68cc38b") : secret "image-registry-tls" not found
Jul 17 21:46:14.358382 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:14.358346    2565 generic.go:356] "Generic (PLEG): container finished" podID="d604d7d6-843e-47f3-b837-31545356dbf3" containerID="5afac2fc7ed4cb159194926c19c0d6a01ef063bbca35f0b5d4969d971888531a" exitCode=0
Jul 17 21:46:14.358559 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:14.358430    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2lwh" event={"ID":"d604d7d6-843e-47f3-b837-31545356dbf3","Type":"ContainerDied","Data":"5afac2fc7ed4cb159194926c19c0d6a01ef063bbca35f0b5d4969d971888531a"}
Jul 17 21:46:14.359545 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:14.359521    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q5vtj" event={"ID":"2c961b41-034e-40be-a3f6-292e81593596","Type":"ContainerStarted","Data":"896c830a67ccf5d3ffedd44ebfe61cef7d7310f20b2b89c2f0a45f4727e344f8"}
Jul 17 21:46:15.237638 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:15.237606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:15.237638 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:15.237649    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:15.238539 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:15.237777    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:46:15.238539 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:15.237785    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:46:15.238539 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:15.237860    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls podName:13be2284-1e6e-414b-bd17-01caea2dc51c nodeName:}" failed. No retries permitted until 2026-07-17 21:46:17.237837696 +0000 UTC m=+40.626854346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls") pod "dns-default-8hzbf" (UID: "13be2284-1e6e-414b-bd17-01caea2dc51c") : secret "dns-default-metrics-tls" not found
Jul 17 21:46:15.238539 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:15.237792    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b875dd99-cs4hm: secret "image-registry-tls" not found
Jul 17 21:46:15.238539 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:15.237962    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls podName:18e6a280-24ab-486f-9870-83b1c68cc38b nodeName:}" failed. No retries permitted until 2026-07-17 21:46:17.237932609 +0000 UTC m=+40.626949239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls") pod "image-registry-54b875dd99-cs4hm" (UID: "18e6a280-24ab-486f-9870-83b1c68cc38b") : secret "image-registry-tls" not found
Jul 17 21:46:15.366244 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:15.366205    2565 generic.go:356] "Generic (PLEG): container finished" podID="d604d7d6-843e-47f3-b837-31545356dbf3" containerID="b85c6e25edfa77755717cd590f3b3caa7da042a77b2705f3f8a9f3f2e2e1b276" exitCode=0
Jul 17 21:46:15.366412 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:15.366267    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2lwh" event={"ID":"d604d7d6-843e-47f3-b837-31545356dbf3","Type":"ContainerDied","Data":"b85c6e25edfa77755717cd590f3b3caa7da042a77b2705f3f8a9f3f2e2e1b276"}
Jul 17 21:46:16.370657 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:16.370616    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q5vtj" event={"ID":"2c961b41-034e-40be-a3f6-292e81593596","Type":"ContainerStarted","Data":"487f6d40e0b8297630ecc9b6b3bdc238645824dfa1eb4b38a10b0797785e8834"}
Jul 17 21:46:16.374282 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:16.374250    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2lwh" event={"ID":"d604d7d6-843e-47f3-b837-31545356dbf3","Type":"ContainerStarted","Data":"7014f9adc2eb66b385c43ca4fca565e21b8e2c1fac261e4023c8f50b6f0736e3"}
Jul 17 21:46:16.385752 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:16.385695    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-q5vtj" podStartSLOduration=1.200260425 podStartE2EDuration="3.385678161s" totalImagesPullingTime="2.185417736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:16.384993139 +0000 UTC m=+39.774009790" watchObservedRunningTime="2026-07-17 21:46:16.385678161 +0000 UTC m=+39.774694811"
Jul 17 21:46:16.407791 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:16.406937    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h2lwh" podStartSLOduration=7.21724709 podStartE2EDuration="39.406914754s" totalImagesPullingTime="32.189667664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:45:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:16.405591101 +0000 UTC m=+39.794607777" watchObservedRunningTime="2026-07-17 21:46:16.406914754 +0000 UTC m=+39.795931422"
Jul 17 21:46:16.526529 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:16.526439    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:16.626802 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:16.626760    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:16.742823 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:16.742592    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:16.801359 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:16.801324    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:17.252092 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:17.252026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:17.252092 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:17.252100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:17.252367 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:17.252199    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:46:17.252367 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:17.252217    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:46:17.252367 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:17.252227    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b875dd99-cs4hm: secret "image-registry-tls" not found
Jul 17 21:46:17.252367 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:17.252270    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls podName:18e6a280-24ab-486f-9870-83b1c68cc38b nodeName:}" failed. No retries permitted until 2026-07-17 21:46:21.252255318 +0000 UTC m=+44.641271947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls") pod "image-registry-54b875dd99-cs4hm" (UID: "18e6a280-24ab-486f-9870-83b1c68cc38b") : secret "image-registry-tls" not found
Jul 17 21:46:17.252367 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:17.252281    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls podName:13be2284-1e6e-414b-bd17-01caea2dc51c nodeName:}" failed. No retries permitted until 2026-07-17 21:46:21.252275806 +0000 UTC m=+44.641292435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls") pod "dns-default-8hzbf" (UID: "13be2284-1e6e-414b-bd17-01caea2dc51c") : secret "dns-default-metrics-tls" not found
Jul 17 21:46:17.458356 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:17.458323    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:17.520359 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:17.520276    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:18.182283 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.182249    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:18.238895 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.238859    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:18.313276 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.313249    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tdvns"]
Jul 17 21:46:18.316280 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.316261    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:18.318827 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.318802    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 21:46:18.319368 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.319350    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 21:46:18.319595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.319582    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 21:46:18.319654 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.319628    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 21:46:18.320421 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.320404    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9pf4b\""
Jul 17 21:46:18.326509 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.326487    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tdvns"]
Jul 17 21:46:18.334269 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.334246    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:18.396340 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.396307    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:18.463014 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.462938    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:18.463014 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.462978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24d59108-63d1-4002-9de4-8cf8287e3992-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:18.463014 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.463007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/24d59108-63d1-4002-9de4-8cf8287e3992-crio-socket\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:18.463470 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.463048    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/24d59108-63d1-4002-9de4-8cf8287e3992-data-volume\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:18.463520 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.463476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lth56\" (UniqueName: \"kubernetes.io/projected/24d59108-63d1-4002-9de4-8cf8287e3992-kube-api-access-lth56\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:18.564540 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.564503    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:18.564540 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.564544    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24d59108-63d1-4002-9de4-8cf8287e3992-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:18.564763 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.564565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/24d59108-63d1-4002-9de4-8cf8287e3992-crio-socket\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:18.564763 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:18.564638    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:46:18.564763 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:18.564687    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:46:18.564763 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:18.564695    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24d59108-63d1-4002-9de4-8cf8287e3992-kube-rbac-proxy-cm podName:24d59108-63d1-4002-9de4-8cf8287e3992 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:19.064678158 +0000 UTC m=+42.453694788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/24d59108-63d1-4002-9de4-8cf8287e3992-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdvns" (UID: "24d59108-63d1-4002-9de4-8cf8287e3992") : configmap "kube-rbac-proxy" not found
Jul 17 21:46:18.564962 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.564762    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/24d59108-63d1-4002-9de4-8cf8287e3992-data-volume\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:18.564962 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:18.564786    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls podName:24d59108-63d1-4002-9de4-8cf8287e3992 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:19.064758697 +0000 UTC m=+42.453775330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdvns" (UID: "24d59108-63d1-4002-9de4-8cf8287e3992") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:46:18.564962 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.564819    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/24d59108-63d1-4002-9de4-8cf8287e3992-crio-socket\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:18.564962 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.564908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lth56\" (UniqueName: \"kubernetes.io/projected/24d59108-63d1-4002-9de4-8cf8287e3992-kube-api-access-lth56\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:18.565126 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.565019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/24d59108-63d1-4002-9de4-8cf8287e3992-data-volume\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:18.577342 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.577311    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lth56\" (UniqueName: \"kubernetes.io/projected/24d59108-63d1-4002-9de4-8cf8287e3992-kube-api-access-lth56\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:18.641536 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.641504    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:18.748718 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.748686    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:18.760294 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.760263    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4ll6f"]
Jul 17 21:46:18.763241 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.763226    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4ll6f"
Jul 17 21:46:18.767111 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.767087    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w5jwg\""
Jul 17 21:46:18.767242 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.767180    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 17 21:46:18.767631 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.767614    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 17 21:46:18.776666 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.776642    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4ll6f"]
Jul 17 21:46:18.844101 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.844052    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:18.867281 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.867254    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2nz4\" (UniqueName: \"kubernetes.io/projected/75819c53-b542-4a7c-af75-49270bed8d13-kube-api-access-p2nz4\") pod \"migrator-7cc9c7ffd7-4ll6f\" (UID: \"75819c53-b542-4a7c-af75-49270bed8d13\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4ll6f"
Jul 17 21:46:18.942731 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.942695    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:18.967857 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.967822    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2nz4\" (UniqueName: \"kubernetes.io/projected/75819c53-b542-4a7c-af75-49270bed8d13-kube-api-access-p2nz4\") pod \"migrator-7cc9c7ffd7-4ll6f\" (UID: \"75819c53-b542-4a7c-af75-49270bed8d13\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4ll6f"
Jul 17 21:46:18.977931 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:18.977906    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2nz4\" (UniqueName: \"kubernetes.io/projected/75819c53-b542-4a7c-af75-49270bed8d13-kube-api-access-p2nz4\") pod \"migrator-7cc9c7ffd7-4ll6f\" (UID: \"75819c53-b542-4a7c-af75-49270bed8d13\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4ll6f"
Jul 17 21:46:19.068850 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:19.068757    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:19.068850 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:19.068798    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24d59108-63d1-4002-9de4-8cf8287e3992-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:19.069044 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:19.068875    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:46:19.069044 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:19.068907    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:46:19.069044 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:19.068925    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24d59108-63d1-4002-9de4-8cf8287e3992-kube-rbac-proxy-cm podName:24d59108-63d1-4002-9de4-8cf8287e3992 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:20.06891044 +0000 UTC m=+43.457927068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/24d59108-63d1-4002-9de4-8cf8287e3992-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdvns" (UID: "24d59108-63d1-4002-9de4-8cf8287e3992") : configmap "kube-rbac-proxy" not found
Jul 17 21:46:19.069044 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:19.069038    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls podName:24d59108-63d1-4002-9de4-8cf8287e3992 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:20.069025448 +0000 UTC m=+43.458042078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdvns" (UID: "24d59108-63d1-4002-9de4-8cf8287e3992") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:46:19.072741 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:19.072723    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4ll6f"
Jul 17 21:46:19.198090 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:19.198043    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4ll6f"]
Jul 17 21:46:19.200902 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:46:19.200877    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75819c53_b542_4a7c_af75_49270bed8d13.slice/crio-fe04557f3ae6cac29e9fc95de11b11994bfd16943b19430c3e2111a24c811d9b WatchSource:0}: Error finding container fe04557f3ae6cac29e9fc95de11b11994bfd16943b19430c3e2111a24c811d9b: Status 404 returned error can't find the container with id fe04557f3ae6cac29e9fc95de11b11994bfd16943b19430c3e2111a24c811d9b
Jul 17 21:46:19.381762 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:19.381681    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4ll6f" event={"ID":"75819c53-b542-4a7c-af75-49270bed8d13","Type":"ContainerStarted","Data":"fe04557f3ae6cac29e9fc95de11b11994bfd16943b19430c3e2111a24c811d9b"}
Jul 17 21:46:20.077737 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.077694    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:20.078165 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.077754    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24d59108-63d1-4002-9de4-8cf8287e3992-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:20.078165 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:20.077848    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:46:20.078165 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:20.077856    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:46:20.078165 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:20.077916    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24d59108-63d1-4002-9de4-8cf8287e3992-kube-rbac-proxy-cm podName:24d59108-63d1-4002-9de4-8cf8287e3992 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:22.077897078 +0000 UTC m=+45.466913722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/24d59108-63d1-4002-9de4-8cf8287e3992-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdvns" (UID: "24d59108-63d1-4002-9de4-8cf8287e3992") : configmap "kube-rbac-proxy" not found
Jul 17 21:46:20.078165 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:20.077937    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls podName:24d59108-63d1-4002-9de4-8cf8287e3992 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:22.077925778 +0000 UTC m=+45.466942415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdvns" (UID: "24d59108-63d1-4002-9de4-8cf8287e3992") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:46:20.106047 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.106013    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sq8rg_79e9c71a-027c-4087-a6e0-1ecf5753a7b5/dns-node-resolver/0.log"
Jul 17 21:46:20.487097 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.487010    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-5pmpc"]
Jul 17 21:46:20.503187 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.503162    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-5pmpc"]
Jul 17 21:46:20.503337 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.503262    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-5pmpc"
Jul 17 21:46:20.505839 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.505813    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 17 21:46:20.505994 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.505850    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 17 21:46:20.505994 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.505822    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 17 21:46:20.506843 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.506819    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 17 21:46:20.506945 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.506855    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5wt6h\""
Jul 17 21:46:20.528593 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.528565    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:20.582407 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.582378    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2ffd87e4-3a66-4957-977d-1eab107d904d-signing-key\") pod \"service-ca-58598695f7-5pmpc\" (UID: \"2ffd87e4-3a66-4957-977d-1eab107d904d\") " pod="openshift-service-ca/service-ca-58598695f7-5pmpc"
Jul 17 21:46:20.582564 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.582494    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4cf\" (UniqueName: \"kubernetes.io/projected/2ffd87e4-3a66-4957-977d-1eab107d904d-kube-api-access-mp4cf\") pod \"service-ca-58598695f7-5pmpc\" (UID: \"2ffd87e4-3a66-4957-977d-1eab107d904d\") " pod="openshift-service-ca/service-ca-58598695f7-5pmpc"
Jul 17 21:46:20.582613 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.582569    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2ffd87e4-3a66-4957-977d-1eab107d904d-signing-cabundle\") pod \"service-ca-58598695f7-5pmpc\" (UID: \"2ffd87e4-3a66-4957-977d-1eab107d904d\") " pod="openshift-service-ca/service-ca-58598695f7-5pmpc"
Jul 17 21:46:20.583241 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.583224    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:20.683574 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.683535    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4cf\" (UniqueName: \"kubernetes.io/projected/2ffd87e4-3a66-4957-977d-1eab107d904d-kube-api-access-mp4cf\") pod \"service-ca-58598695f7-5pmpc\" (UID: \"2ffd87e4-3a66-4957-977d-1eab107d904d\") " pod="openshift-service-ca/service-ca-58598695f7-5pmpc"
Jul 17 21:46:20.683746 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.683594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2ffd87e4-3a66-4957-977d-1eab107d904d-signing-cabundle\") pod \"service-ca-58598695f7-5pmpc\" (UID: \"2ffd87e4-3a66-4957-977d-1eab107d904d\") " pod="openshift-service-ca/service-ca-58598695f7-5pmpc"
Jul 17 21:46:20.683746 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.683629    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2ffd87e4-3a66-4957-977d-1eab107d904d-signing-key\") pod \"service-ca-58598695f7-5pmpc\" (UID: \"2ffd87e4-3a66-4957-977d-1eab107d904d\") " pod="openshift-service-ca/service-ca-58598695f7-5pmpc"
Jul 17 21:46:20.684340 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.684317    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2ffd87e4-3a66-4957-977d-1eab107d904d-signing-cabundle\") pod \"service-ca-58598695f7-5pmpc\" (UID: \"2ffd87e4-3a66-4957-977d-1eab107d904d\") " pod="openshift-service-ca/service-ca-58598695f7-5pmpc"
Jul 17 21:46:20.686135 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.686116    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2ffd87e4-3a66-4957-977d-1eab107d904d-signing-key\") pod \"service-ca-58598695f7-5pmpc\" (UID: \"2ffd87e4-3a66-4957-977d-1eab107d904d\") " pod="openshift-service-ca/service-ca-58598695f7-5pmpc"
Jul 17 21:46:20.691996 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.691973    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4cf\" (UniqueName: \"kubernetes.io/projected/2ffd87e4-3a66-4957-977d-1eab107d904d-kube-api-access-mp4cf\") pod \"service-ca-58598695f7-5pmpc\" (UID: \"2ffd87e4-3a66-4957-977d-1eab107d904d\") " pod="openshift-service-ca/service-ca-58598695f7-5pmpc"
Jul 17 21:46:20.812902 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.812851    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-5pmpc"
Jul 17 21:46:20.910322 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.909932    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ls9f6_b77bccab-4a6d-4e18-8206-9ad86e070317/node-ca/0.log"
Jul 17 21:46:20.959570 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:20.959541    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-5pmpc"]
Jul 17 21:46:20.962631 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:46:20.962611    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffd87e4_3a66_4957_977d_1eab107d904d.slice/crio-b5116588d630bce025d4f6d28c5d144bd3cccabc3dfc4faa04f4f3357c5ebdc0 WatchSource:0}: Error finding container b5116588d630bce025d4f6d28c5d144bd3cccabc3dfc4faa04f4f3357c5ebdc0: Status 404 returned error can't find the container with id b5116588d630bce025d4f6d28c5d144bd3cccabc3dfc4faa04f4f3357c5ebdc0
Jul 17 21:46:21.288670 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:21.288635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:21.289017 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:21.288677    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:21.289017 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:21.288786    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:46:21.289017 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:21.288787    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:46:21.289017 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:21.288796    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54b875dd99-cs4hm: secret "image-registry-tls" not found
Jul 17 21:46:21.289017 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:21.288856    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls podName:13be2284-1e6e-414b-bd17-01caea2dc51c nodeName:}" failed. No retries permitted until 2026-07-17 21:46:29.288840537 +0000 UTC m=+52.677857165 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls") pod "dns-default-8hzbf" (UID: "13be2284-1e6e-414b-bd17-01caea2dc51c") : secret "dns-default-metrics-tls" not found
Jul 17 21:46:21.289017 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:21.288870    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls podName:18e6a280-24ab-486f-9870-83b1c68cc38b nodeName:}" failed. No retries permitted until 2026-07-17 21:46:29.28886451 +0000 UTC m=+52.677881139 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls") pod "image-registry-54b875dd99-cs4hm" (UID: "18e6a280-24ab-486f-9870-83b1c68cc38b") : secret "image-registry-tls" not found
Jul 17 21:46:21.391384 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:21.391349    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4ll6f" event={"ID":"75819c53-b542-4a7c-af75-49270bed8d13","Type":"ContainerStarted","Data":"550c3d1e17e9a54f0d3dff89dddac254a50917e56c2aa74892b70c55aa5b52d7"}
Jul 17 21:46:21.391578 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:21.391445    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4ll6f" event={"ID":"75819c53-b542-4a7c-af75-49270bed8d13","Type":"ContainerStarted","Data":"8e2058ffeeffa4b8e3e0cd25ff914325b856a233a717fd08679eb13e27eb77c6"}
Jul 17 21:46:21.392568 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:21.392531    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-5pmpc" event={"ID":"2ffd87e4-3a66-4957-977d-1eab107d904d","Type":"ContainerStarted","Data":"b5116588d630bce025d4f6d28c5d144bd3cccabc3dfc4faa04f4f3357c5ebdc0"}
Jul 17 21:46:21.411711 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:21.411660    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4ll6f" podStartSLOduration=1.891069312 podStartE2EDuration="3.411645406s" totalImagesPullingTime="1.520576094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:21.411113138 +0000 UTC m=+44.800129788" watchObservedRunningTime="2026-07-17 21:46:21.411645406 +0000 UTC m=+44.800662056"
Jul 17 21:46:21.514054 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:21.513999    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:21.590829 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:21.590734    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:21.705605 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:21.705579    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q5vtj_2c961b41-034e-40be-a3f6-292e81593596/serve-healthcheck-canary/0.log"
Jul 17 21:46:22.096024 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:22.095979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:22.096024 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:22.096037    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24d59108-63d1-4002-9de4-8cf8287e3992-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:22.096314 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:22.096197    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:46:22.096314 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:22.096208    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:46:22.096314 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:22.096280    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls podName:24d59108-63d1-4002-9de4-8cf8287e3992 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:26.096257956 +0000 UTC m=+49.485274628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdvns" (UID: "24d59108-63d1-4002-9de4-8cf8287e3992") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:46:22.096314 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:22.096310    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24d59108-63d1-4002-9de4-8cf8287e3992-kube-rbac-proxy-cm podName:24d59108-63d1-4002-9de4-8cf8287e3992 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:26.096298921 +0000 UTC m=+49.485315555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/24d59108-63d1-4002-9de4-8cf8287e3992-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdvns" (UID: "24d59108-63d1-4002-9de4-8cf8287e3992") : configmap "kube-rbac-proxy" not found
Jul 17 21:46:23.399503 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:23.399413    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-5pmpc" event={"ID":"2ffd87e4-3a66-4957-977d-1eab107d904d","Type":"ContainerStarted","Data":"96f56141627fb0e8da7317a65b31830a10043d62e31573881874bdf1087f88b2"}
Jul 17 21:46:23.418259 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:23.418203    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-5pmpc" podStartSLOduration=1.237940952 podStartE2EDuration="3.418187952s" totalImagesPullingTime="2.180247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:23.417213915 +0000 UTC m=+46.806230586" watchObservedRunningTime="2026-07-17 21:46:23.418187952 +0000 UTC m=+46.807204604"
Jul 17 21:46:23.942485 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:23.942439    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:24.648962 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:24.648903    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:24.752692 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:24.752657    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:24.849560 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:24.849519    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:24.960007 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:24.959895    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:25.060778 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:25.060731    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:25.161740 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:25.161707    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:25.474667 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:25.474627    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:25.575845 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:25.575802    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:25.674856 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:25.674821    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:25.775587 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:25.775556    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:25.878691 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:25.878651    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:26.127677 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:26.127602    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:26.127677 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:26.127637    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24d59108-63d1-4002-9de4-8cf8287e3992-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:26.127862 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:26.127764    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:46:26.127862 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:46:26.127836    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls podName:24d59108-63d1-4002-9de4-8cf8287e3992 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:34.127821092 +0000 UTC m=+57.516837724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdvns" (UID: "24d59108-63d1-4002-9de4-8cf8287e3992") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:46:26.128153 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:26.128137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24d59108-63d1-4002-9de4-8cf8287e3992-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:27.022598 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:27.022566    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:27.227938 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:27.227884    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:27.439001 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:27.438911    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:27.619152 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:27.619112    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:27.825142 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:27.825094    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:28.016031 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:28.016003    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:28.620410 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:28.620378    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:29.027552 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:29.027515    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:29.225575 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:29.225541    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:29.359308 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:29.359213    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:29.359308 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:29.359258    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:29.361824 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:29.361791    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13be2284-1e6e-414b-bd17-01caea2dc51c-metrics-tls\") pod \"dns-default-8hzbf\" (UID: \"13be2284-1e6e-414b-bd17-01caea2dc51c\") " pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:29.361933 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:29.361844    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls\") pod \"image-registry-54b875dd99-cs4hm\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") " pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:29.395025 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:29.394992    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:29.416180 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:29.416136    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:29.421093 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:29.421053    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:29.518469 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:29.518429    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:29.568890 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:29.568861    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54b875dd99-cs4hm"]
Jul 17 21:46:29.572623 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:46:29.572591    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e6a280_24ab_486f_9870_83b1c68cc38b.slice/crio-aa17292cfa598c58959b280a9fb3de3407e3ac51b3c9446e217c36498685687d WatchSource:0}: Error finding container aa17292cfa598c58959b280a9fb3de3407e3ac51b3c9446e217c36498685687d: Status 404 returned error can't find the container with id aa17292cfa598c58959b280a9fb3de3407e3ac51b3c9446e217c36498685687d
Jul 17 21:46:29.584699 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:29.584675    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8hzbf"]
Jul 17 21:46:29.587882 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:46:29.587845    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13be2284_1e6e_414b_bd17_01caea2dc51c.slice/crio-018abaf236e1db4a97688c3642b1c11bc9818a4c430436e3404230d41aefcb13 WatchSource:0}: Error finding container 018abaf236e1db4a97688c3642b1c11bc9818a4c430436e3404230d41aefcb13: Status 404 returned error can't find the container with id 018abaf236e1db4a97688c3642b1c11bc9818a4c430436e3404230d41aefcb13
Jul 17 21:46:29.588219 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:29.588179    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:29.645867 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:29.645842    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:29.747713 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:29.747678    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:29.846741 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:29.846709    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:30.097482 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:30.097438    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:30.158164 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:30.158129    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:30.420535 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:30.420442    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8hzbf" event={"ID":"13be2284-1e6e-414b-bd17-01caea2dc51c","Type":"ContainerStarted","Data":"018abaf236e1db4a97688c3642b1c11bc9818a4c430436e3404230d41aefcb13"}
Jul 17 21:46:30.422280 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:30.422245    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54b875dd99-cs4hm" event={"ID":"18e6a280-24ab-486f-9870-83b1c68cc38b","Type":"ContainerStarted","Data":"228a8a67d2c9829792889f63ab3498396e079ee53a3643f43dc31f41a0828765"}
Jul 17 21:46:30.422427 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:30.422296    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54b875dd99-cs4hm" event={"ID":"18e6a280-24ab-486f-9870-83b1c68cc38b","Type":"ContainerStarted","Data":"aa17292cfa598c58959b280a9fb3de3407e3ac51b3c9446e217c36498685687d"}
Jul 17 21:46:30.422427 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:30.422332    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:30.443825 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:30.443756    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54b875dd99-cs4hm" podStartSLOduration=33.44373646 podStartE2EDuration="33.44373646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:30.442338223 +0000 UTC m=+53.831354887" watchObservedRunningTime="2026-07-17 21:46:30.44373646 +0000 UTC m=+53.832753095"
Jul 17 21:46:30.535049 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:30.535004    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:30.636005 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:30.635968    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:30.825651 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:30.825617    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:30.926392 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:30.926358    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:31.111313 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:31.111229    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:31.223352 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:31.223316    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:31.428226 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:31.428141    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:31.631304 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:31.631267    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:31.811838 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:31.811807    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:31.923149 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:31.923100    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:32.029540 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:32.029516    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:32.430469 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:32.430380    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8hzbf" event={"ID":"13be2284-1e6e-414b-bd17-01caea2dc51c","Type":"ContainerStarted","Data":"16c794389c659c7b95fa587264848d26597eb1472c855af939ef8ddec26d7e77"}
Jul 17 21:46:32.430469 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:32.430439    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8hzbf" event={"ID":"13be2284-1e6e-414b-bd17-01caea2dc51c","Type":"ContainerStarted","Data":"6a2a511120d6d5c6a9b953803c8db46aa716711b8bd27b17df73db7506160a55"}
Jul 17 21:46:32.430664 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:32.430475    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:32.446896 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:32.446845    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8hzbf" podStartSLOduration=17.094496294 podStartE2EDuration="19.446827736s" totalImagesPullingTime="2.352331442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:32.446469765 +0000 UTC m=+55.835486416" watchObservedRunningTime="2026-07-17 21:46:32.446827736 +0000 UTC m=+55.835844388"
Jul 17 21:46:32.620466 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:32.620391    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:32.973842 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:32.973809    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:33.074551 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:33.074509    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:33.420365 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:33.420327    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:33.622805 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:33.622769    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:34.029698 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:34.029662    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:34.196232 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:34.196177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:34.198711 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:34.198679    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24d59108-63d1-4002-9de4-8cf8287e3992-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdvns\" (UID: \"24d59108-63d1-4002-9de4-8cf8287e3992\") " pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:34.225729 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:34.225684    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tdvns"
Jul 17 21:46:34.374776 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:34.374743    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tdvns"]
Jul 17 21:46:34.378169 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:46:34.378142    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d59108_63d1_4002_9de4_8cf8287e3992.slice/crio-121ff3f69551cb0ea5f6661023d8f2ba808cd7437bd78a09258ef3078e01681d WatchSource:0}: Error finding container 121ff3f69551cb0ea5f6661023d8f2ba808cd7437bd78a09258ef3078e01681d: Status 404 returned error can't find the container with id 121ff3f69551cb0ea5f6661023d8f2ba808cd7437bd78a09258ef3078e01681d
Jul 17 21:46:34.438128 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:34.438096    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdvns" event={"ID":"24d59108-63d1-4002-9de4-8cf8287e3992","Type":"ContainerStarted","Data":"121ff3f69551cb0ea5f6661023d8f2ba808cd7437bd78a09258ef3078e01681d"}
Jul 17 21:46:34.640314 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:34.640201    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:35.054548 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:35.054507    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:35.433654 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:35.433573    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:35.641003 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:35.640962    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:35.894224 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:35.894183    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:38.352012 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:38.351981    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bhftj"
Jul 17 21:46:39.454426 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:39.454385    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdvns" event={"ID":"24d59108-63d1-4002-9de4-8cf8287e3992","Type":"ContainerStarted","Data":"8a8d90c0c56ae689bfcb462ae509029cf2e46fc2961795e34b12fd6d0415ad41"}
Jul 17 21:46:40.286097 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.285950    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-22npp"]
Jul 17 21:46:40.289147 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.289120    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-22npp"
Jul 17 21:46:40.292129 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.291780    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 17 21:46:40.292129 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.291974    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fx6g7\""
Jul 17 21:46:40.292823 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.292699    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 17 21:46:40.302215 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.302147    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-22npp"]
Jul 17 21:46:40.307737 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.307712    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54b875dd99-cs4hm"]
Jul 17 21:46:40.348329 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.348297    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/13fc622a-3c51-485a-8b45-1ee687ceeb17-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-22npp\" (UID: \"13fc622a-3c51-485a-8b45-1ee687ceeb17\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-22npp"
Jul 17 21:46:40.348523 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.348374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13fc622a-3c51-485a-8b45-1ee687ceeb17-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-22npp\" (UID: \"13fc622a-3c51-485a-8b45-1ee687ceeb17\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-22npp"
Jul 17 21:46:40.388319 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.387571    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-8p6dp"]
Jul 17 21:46:40.390367 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.390340    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55498b58b5-9kpmt"]
Jul 17 21:46:40.390503 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.390460    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-8p6dp"
Jul 17 21:46:40.392943 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.392912    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:40.393565 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.393546    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.395560 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.395346    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 17 21:46:40.395560 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.395346    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 17 21:46:40.395560 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.395374    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-jl6vx\""
Jul 17 21:46:40.408254 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.408232    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-8p6dp"]
Jul 17 21:46:40.411009 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.410986    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55498b58b5-9kpmt"]
Jul 17 21:46:40.448837 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.448799    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxgzp\" (UniqueName: \"kubernetes.io/projected/f8504435-3631-4552-8ed1-70376a9ff7b4-kube-api-access-jxgzp\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.448837 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.448843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8504435-3631-4552-8ed1-70376a9ff7b4-registry-certificates\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.449118 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.448888    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8504435-3631-4552-8ed1-70376a9ff7b4-bound-sa-token\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.449118 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.448909    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8504435-3631-4552-8ed1-70376a9ff7b4-registry-tls\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.449118 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.448973    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxgmt\" (UniqueName: \"kubernetes.io/projected/7929fd6d-36e0-46bf-83e6-7a93999d2f01-kube-api-access-mxgmt\") pod \"downloads-5bc67dcd46-8p6dp\" (UID: \"7929fd6d-36e0-46bf-83e6-7a93999d2f01\") " pod="openshift-console/downloads-5bc67dcd46-8p6dp"
Jul 17 21:46:40.449118 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.449004    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13fc622a-3c51-485a-8b45-1ee687ceeb17-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-22npp\" (UID: \"13fc622a-3c51-485a-8b45-1ee687ceeb17\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-22npp"
Jul 17 21:46:40.449118 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.449100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/13fc622a-3c51-485a-8b45-1ee687ceeb17-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-22npp\" (UID: \"13fc622a-3c51-485a-8b45-1ee687ceeb17\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-22npp"
Jul 17 21:46:40.449386 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.449145    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8504435-3631-4552-8ed1-70376a9ff7b4-installation-pull-secrets\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.449386 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.449171    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8504435-3631-4552-8ed1-70376a9ff7b4-ca-trust-extracted\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.449386 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.449200    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8504435-3631-4552-8ed1-70376a9ff7b4-image-registry-private-configuration\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.449386 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.449223    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8504435-3631-4552-8ed1-70376a9ff7b4-trusted-ca\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.450193 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.450169    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/13fc622a-3c51-485a-8b45-1ee687ceeb17-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-22npp\" (UID: \"13fc622a-3c51-485a-8b45-1ee687ceeb17\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-22npp"
Jul 17 21:46:40.451717 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.451698    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13fc622a-3c51-485a-8b45-1ee687ceeb17-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-22npp\" (UID: \"13fc622a-3c51-485a-8b45-1ee687ceeb17\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-22npp"
Jul 17 21:46:40.469872 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.469839    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:40.550120 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.550009    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8504435-3631-4552-8ed1-70376a9ff7b4-bound-sa-token\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.550120 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.550059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8504435-3631-4552-8ed1-70376a9ff7b4-registry-tls\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.550340 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.550125    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxgmt\" (UniqueName: \"kubernetes.io/projected/7929fd6d-36e0-46bf-83e6-7a93999d2f01-kube-api-access-mxgmt\") pod \"downloads-5bc67dcd46-8p6dp\" (UID: \"7929fd6d-36e0-46bf-83e6-7a93999d2f01\") " pod="openshift-console/downloads-5bc67dcd46-8p6dp"
Jul 17 21:46:40.550340 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.550212    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8504435-3631-4552-8ed1-70376a9ff7b4-installation-pull-secrets\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.550340 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.550235    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8504435-3631-4552-8ed1-70376a9ff7b4-ca-trust-extracted\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.550340 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.550257    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8504435-3631-4552-8ed1-70376a9ff7b4-image-registry-private-configuration\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.550340 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.550272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8504435-3631-4552-8ed1-70376a9ff7b4-trusted-ca\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.550340 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.550300    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxgzp\" (UniqueName: \"kubernetes.io/projected/f8504435-3631-4552-8ed1-70376a9ff7b4-kube-api-access-jxgzp\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.550340 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.550331    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8504435-3631-4552-8ed1-70376a9ff7b4-registry-certificates\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.550770 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.550748    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8504435-3631-4552-8ed1-70376a9ff7b4-ca-trust-extracted\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.551649 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.551590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8504435-3631-4552-8ed1-70376a9ff7b4-registry-certificates\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.551758 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.551685    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8504435-3631-4552-8ed1-70376a9ff7b4-trusted-ca\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.553037 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.553015    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8504435-3631-4552-8ed1-70376a9ff7b4-registry-tls\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.553767 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.553743    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8504435-3631-4552-8ed1-70376a9ff7b4-installation-pull-secrets\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.553904 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.553886    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8504435-3631-4552-8ed1-70376a9ff7b4-image-registry-private-configuration\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.561304 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.561280    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8504435-3631-4552-8ed1-70376a9ff7b4-bound-sa-token\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.561560 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.561535    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxgzp\" (UniqueName: \"kubernetes.io/projected/f8504435-3631-4552-8ed1-70376a9ff7b4-kube-api-access-jxgzp\") pod \"image-registry-55498b58b5-9kpmt\" (UID: \"f8504435-3631-4552-8ed1-70376a9ff7b4\") " pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.562985 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.562964    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxgmt\" (UniqueName: \"kubernetes.io/projected/7929fd6d-36e0-46bf-83e6-7a93999d2f01-kube-api-access-mxgmt\") pod \"downloads-5bc67dcd46-8p6dp\" (UID: \"7929fd6d-36e0-46bf-83e6-7a93999d2f01\") " pod="openshift-console/downloads-5bc67dcd46-8p6dp"
Jul 17 21:46:40.601685 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.601651    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-22npp"
Jul 17 21:46:40.701948 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.701702    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-8p6dp"
Jul 17 21:46:40.709113 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.709055    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:40.746846 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.746727    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-22npp"]
Jul 17 21:46:40.750229 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:46:40.750174    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fc622a_3c51_485a_8b45_1ee687ceeb17.slice/crio-d9336b8031df887839a737dc7d7021f1379d028a8ca343dd6ad9ef96126ef2a4 WatchSource:0}: Error finding container d9336b8031df887839a737dc7d7021f1379d028a8ca343dd6ad9ef96126ef2a4: Status 404 returned error can't find the container with id d9336b8031df887839a737dc7d7021f1379d028a8ca343dd6ad9ef96126ef2a4
Jul 17 21:46:40.750945 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.750517    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:40.812865 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.812779    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:40.862504 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.862428    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-8p6dp"]
Jul 17 21:46:40.864591 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:46:40.864559    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7929fd6d_36e0_46bf_83e6_7a93999d2f01.slice/crio-ee65fd11d01e5496831929f96528c753f1e1da26d39a1a5c3d7474acabbc9ac3 WatchSource:0}: Error finding container ee65fd11d01e5496831929f96528c753f1e1da26d39a1a5c3d7474acabbc9ac3: Status 404 returned error can't find the container with id ee65fd11d01e5496831929f96528c753f1e1da26d39a1a5c3d7474acabbc9ac3
Jul 17 21:46:40.886383 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:40.886351    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55498b58b5-9kpmt"]
Jul 17 21:46:40.889842 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:46:40.889814    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8504435_3631_4552_8ed1_70376a9ff7b4.slice/crio-cad608598269365c3f7844434a4ffbd49a81e1b8b72e97e35ead6f601b61bad4 WatchSource:0}: Error finding container cad608598269365c3f7844434a4ffbd49a81e1b8b72e97e35ead6f601b61bad4: Status 404 returned error can't find the container with id cad608598269365c3f7844434a4ffbd49a81e1b8b72e97e35ead6f601b61bad4
Jul 17 21:46:41.463177 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.463105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55498b58b5-9kpmt" event={"ID":"f8504435-3631-4552-8ed1-70376a9ff7b4","Type":"ContainerStarted","Data":"6337e1183ed44f23c6b9a588a0ae5b3f9c575aa281e4d3e00520128cf7403698"}
Jul 17 21:46:41.463177 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.463167    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55498b58b5-9kpmt" event={"ID":"f8504435-3631-4552-8ed1-70376a9ff7b4","Type":"ContainerStarted","Data":"cad608598269365c3f7844434a4ffbd49a81e1b8b72e97e35ead6f601b61bad4"}
Jul 17 21:46:41.463438 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.463316    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:46:41.464825 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.464788    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-22npp" event={"ID":"13fc622a-3c51-485a-8b45-1ee687ceeb17","Type":"ContainerStarted","Data":"d9336b8031df887839a737dc7d7021f1379d028a8ca343dd6ad9ef96126ef2a4"}
Jul 17 21:46:41.465977 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.465958    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-8p6dp" event={"ID":"7929fd6d-36e0-46bf-83e6-7a93999d2f01","Type":"ContainerStarted","Data":"ee65fd11d01e5496831929f96528c753f1e1da26d39a1a5c3d7474acabbc9ac3"}
Jul 17 21:46:41.485858 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.485795    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55498b58b5-9kpmt" podStartSLOduration=1.485771695 podStartE2EDuration="1.485771695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:41.483991552 +0000 UTC m=+64.873008198" watchObservedRunningTime="2026-07-17 21:46:41.485771695 +0000 UTC m=+64.874788351"
Jul 17 21:46:41.548358 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.548274    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:41.798414 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.798342    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:41.862396 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.862364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs\") pod \"network-metrics-daemon-cd95t\" (UID: \"ab648b99-a119-46ea-894c-2c83d2658261\") " pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:46:41.864782 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.864759    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 21:46:41.875883 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.875844    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab648b99-a119-46ea-894c-2c83d2658261-metrics-certs\") pod \"network-metrics-daemon-cd95t\" (UID: \"ab648b99-a119-46ea-894c-2c83d2658261\") " pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:46:41.894031 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.893999    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:41.963484 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.963444    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x456\" (UniqueName: \"kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456\") pod \"network-check-target-mg8bv\" (UID: \"7e18d002-d4f9-49ff-b425-fc3c56585a80\") " pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:46:41.963677 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.963569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret\") pod \"global-pull-secret-syncer-hf7sc\" (UID: \"aeb396b3-2508-4c93-aeee-188c1b39fa9d\") " pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:46:41.966248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.966220    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 21:46:41.966402 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.966221    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 21:46:41.976941 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.976915    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 21:46:41.977291 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.977260    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb396b3-2508-4c93-aeee-188c1b39fa9d-original-pull-secret\") pod \"global-pull-secret-syncer-hf7sc\" (UID: \"aeb396b3-2508-4c93-aeee-188c1b39fa9d\") " pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:46:41.987364 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:41.987339    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x456\" (UniqueName: \"kubernetes.io/projected/7e18d002-d4f9-49ff-b425-fc3c56585a80-kube-api-access-2x456\") pod \"network-check-target-mg8bv\" (UID: \"7e18d002-d4f9-49ff-b425-fc3c56585a80\") " pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:46:42.109661 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:42.109580    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mbv8k\""
Jul 17 21:46:42.114833 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:42.114806    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-28dc9\""
Jul 17 21:46:42.117695 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:42.117673    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cd95t"
Jul 17 21:46:42.117854 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:42.117736    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hf7sc"
Jul 17 21:46:42.123970 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:42.123927    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:46:42.436672 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:42.436595    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8hzbf"
Jul 17 21:46:43.240187 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:43.240154    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hf7sc"]
Jul 17 21:46:43.486512 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:46:43.486433    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb396b3_2508_4c93_aeee_188c1b39fa9d.slice/crio-692afaadfaa0abba04e6ffc83ad671a03ddbe354013f7d9c31a930a79797c9b4 WatchSource:0}: Error finding container 692afaadfaa0abba04e6ffc83ad671a03ddbe354013f7d9c31a930a79797c9b4: Status 404 returned error can't find the container with id 692afaadfaa0abba04e6ffc83ad671a03ddbe354013f7d9c31a930a79797c9b4
Jul 17 21:46:43.643178 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:43.643148    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mg8bv"]
Jul 17 21:46:43.645331 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:46:43.645295    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e18d002_d4f9_49ff_b425_fc3c56585a80.slice/crio-111b01c2961bd78244b7938683f23a957b04eba8997693dac28fd128699a8cd9 WatchSource:0}: Error finding container 111b01c2961bd78244b7938683f23a957b04eba8997693dac28fd128699a8cd9: Status 404 returned error can't find the container with id 111b01c2961bd78244b7938683f23a957b04eba8997693dac28fd128699a8cd9
Jul 17 21:46:43.673339 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:43.673303    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cd95t"]
Jul 17 21:46:43.676797 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:46:43.676770    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab648b99_a119_46ea_894c_2c83d2658261.slice/crio-ce0d7f32c09c9dd571a38697bd49e04a342a63ceb37f2633bf9f626c045b761e WatchSource:0}: Error finding container ce0d7f32c09c9dd571a38697bd49e04a342a63ceb37f2633bf9f626c045b761e: Status 404 returned error can't find the container with id ce0d7f32c09c9dd571a38697bd49e04a342a63ceb37f2633bf9f626c045b761e
Jul 17 21:46:44.022710 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.022674    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:44.123627 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.123585    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:44.480817 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.480714    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cd95t" event={"ID":"ab648b99-a119-46ea-894c-2c83d2658261","Type":"ContainerStarted","Data":"ce0d7f32c09c9dd571a38697bd49e04a342a63ceb37f2633bf9f626c045b761e"}
Jul 17 21:46:44.482657 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.482618    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mg8bv" event={"ID":"7e18d002-d4f9-49ff-b425-fc3c56585a80","Type":"ContainerStarted","Data":"111b01c2961bd78244b7938683f23a957b04eba8997693dac28fd128699a8cd9"}
Jul 17 21:46:44.485130 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.485038    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hf7sc" event={"ID":"aeb396b3-2508-4c93-aeee-188c1b39fa9d","Type":"ContainerStarted","Data":"692afaadfaa0abba04e6ffc83ad671a03ddbe354013f7d9c31a930a79797c9b4"}
Jul 17 21:46:44.487645 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.487554    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdvns" event={"ID":"24d59108-63d1-4002-9de4-8cf8287e3992","Type":"ContainerStarted","Data":"88984f28a55732b2f55d979719d4b28d984801f2a7e402936ae6ccd1e9c42795"}
Jul 17 21:46:44.490267 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.490242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-22npp" event={"ID":"13fc622a-3c51-485a-8b45-1ee687ceeb17","Type":"ContainerStarted","Data":"cb398869049d487fdbd846ac0b6d877fc534f773ba5c0466768442014409bbf3"}
Jul 17 21:46:44.520952 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.520892    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-22npp" podStartSLOduration=1.749360619 podStartE2EDuration="4.520870517s" totalImagesPullingTime="2.771509898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:44.518872084 +0000 UTC m=+67.907888761" watchObservedRunningTime="2026-07-17 21:46:44.520870517 +0000 UTC m=+67.909887171"
Jul 17 21:46:44.628937 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.628900    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64b4b5b48-wgpvl"]
Jul 17 21:46:44.631821 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.631733    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:44.633167 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.632410    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.636092 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.635799    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 17 21:46:44.636241 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.636139    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 17 21:46:44.637440 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.636643    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 17 21:46:44.637440 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.636735    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 17 21:46:44.637440 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.636888    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 17 21:46:44.637440 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.637257    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hlhgr\""
Jul 17 21:46:44.659448 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.659368    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64b4b5b48-wgpvl"]
Jul 17 21:46:44.687629 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.685521    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltp92\" (UniqueName: \"kubernetes.io/projected/8658b73d-a8e3-4934-bca6-26e294bf7000-kube-api-access-ltp92\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.687629 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.685569    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-oauth-serving-cert\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.687629 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.685600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-service-ca\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.687629 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.685632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8658b73d-a8e3-4934-bca6-26e294bf7000-console-oauth-config\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.687629 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.685669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-console-config\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.687629 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.685721    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8658b73d-a8e3-4934-bca6-26e294bf7000-console-serving-cert\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.732174 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.732024    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:44.789000 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.786539    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltp92\" (UniqueName: \"kubernetes.io/projected/8658b73d-a8e3-4934-bca6-26e294bf7000-kube-api-access-ltp92\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.789000 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.786590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-oauth-serving-cert\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.789000 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.786623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-service-ca\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.789000 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.786655    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8658b73d-a8e3-4934-bca6-26e294bf7000-console-oauth-config\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.789000 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.786680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-console-config\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.789000 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.786722    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8658b73d-a8e3-4934-bca6-26e294bf7000-console-serving-cert\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.789000 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.787842    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-service-ca\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.789000 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.788690    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-oauth-serving-cert\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.790172 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.790119    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-console-config\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.792758 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.792721    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8658b73d-a8e3-4934-bca6-26e294bf7000-console-oauth-config\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.799851 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.799326    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltp92\" (UniqueName: \"kubernetes.io/projected/8658b73d-a8e3-4934-bca6-26e294bf7000-kube-api-access-ltp92\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.799851 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.799807    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8658b73d-a8e3-4934-bca6-26e294bf7000-console-serving-cert\") pod \"console-64b4b5b48-wgpvl\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") " pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:44.815574 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.815488    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:44.949279 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:44.948913    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:45.164787 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:45.164748    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64b4b5b48-wgpvl"]
Jul 17 21:46:45.489720 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:46:45.489644    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8658b73d_a8e3_4934_bca6_26e294bf7000.slice/crio-16b34ecd58e01e6c4df108db1f2ffc6dccd1ea7084850f6bce24f8a7a0b49a4f WatchSource:0}: Error finding container 16b34ecd58e01e6c4df108db1f2ffc6dccd1ea7084850f6bce24f8a7a0b49a4f: Status 404 returned error can't find the container with id 16b34ecd58e01e6c4df108db1f2ffc6dccd1ea7084850f6bce24f8a7a0b49a4f
Jul 17 21:46:45.501811 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:45.501782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b4b5b48-wgpvl" event={"ID":"8658b73d-a8e3-4934-bca6-26e294bf7000","Type":"ContainerStarted","Data":"16b34ecd58e01e6c4df108db1f2ffc6dccd1ea7084850f6bce24f8a7a0b49a4f"}
Jul 17 21:46:45.941023 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:45.940899    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:46.037027 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:46.036931    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:46.189686 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:46.189611    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:46.270042 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:46.270001    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:46.510992 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:46.510955    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cd95t" event={"ID":"ab648b99-a119-46ea-894c-2c83d2658261","Type":"ContainerStarted","Data":"b4e1ab61c21dee3433d2725c35e65507af0bc1292fc126cd36e379fee0da2745"}
Jul 17 21:46:46.511477 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:46.511020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cd95t" event={"ID":"ab648b99-a119-46ea-894c-2c83d2658261","Type":"ContainerStarted","Data":"d45958823351fe3f04cb089dadbf2e7d845bde91c2ba52cb3c530ea5757a13c3"}
Jul 17 21:46:46.529577 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:46.529518    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cd95t" podStartSLOduration=67.660692343 podStartE2EDuration="1m9.529496804s" totalImagesPullingTime="1.868804461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:46.528474322 +0000 UTC m=+69.917490973" watchObservedRunningTime="2026-07-17 21:46:46.529496804 +0000 UTC m=+69.918513456"
Jul 17 21:46:46.640826 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:46.640793    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:46.741262 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:46.741228    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:47.567039 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:47.566922    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:47.637453 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:47.637343    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:48.834316 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:48.834257    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:48.899882 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:48.899852    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:49.199316 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:49.199229    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:49.345355 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:49.345315    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:49.449218 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:49.449172    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:50.315226 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:50.315161    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:46:51.588001 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:51.587954    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:51.650422 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:51.650387    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:51.734023 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:51.733974    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:51.836826 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:51.836790    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:51.934354 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:51.934228    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:52.744645 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:52.744613    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:52.852378 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:52.852342    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:53.549431 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:53.549382    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdvns" event={"ID":"24d59108-63d1-4002-9de4-8cf8287e3992","Type":"ContainerStarted","Data":"ab02df6f304f042a7b6790a36f98bafb920136e8d6fb9f8ee65b8dd632bc058f"}
Jul 17 21:46:53.551109 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:53.551079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b4b5b48-wgpvl" event={"ID":"8658b73d-a8e3-4934-bca6-26e294bf7000","Type":"ContainerStarted","Data":"b60eb1dad5771b897e8a2a43a3df7023762c809861d08418c28ffe8209d6dd87"}
Jul 17 21:46:53.552672 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:53.552641    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mg8bv" event={"ID":"7e18d002-d4f9-49ff-b425-fc3c56585a80","Type":"ContainerStarted","Data":"219774bd56cdd519f6cb943e15cf1071eebf1f6000d97fd092015ca406c5bc54"}
Jul 17 21:46:53.553037 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:53.553017    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:46:53.554283 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:53.554252    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hf7sc" event={"ID":"aeb396b3-2508-4c93-aeee-188c1b39fa9d","Type":"ContainerStarted","Data":"bb3df0e3e4426ca96e5ed6547cfd798b6fe5557f0672d36c87961313e439c7fb"}
Jul 17 21:46:53.571917 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:53.571823    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tdvns" podStartSLOduration=16.761249747 podStartE2EDuration="35.571802752s" totalImagesPullingTime="18.810553005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:46:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:53.570941782 +0000 UTC m=+76.959958432" watchObservedRunningTime="2026-07-17 21:46:53.571802752 +0000 UTC m=+76.960819403"
Jul 17 21:46:53.587985 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:53.587937    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mg8bv" podStartSLOduration=66.885593316 podStartE2EDuration="1m16.587918134s" totalImagesPullingTime="9.702324818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:53.58729516 +0000 UTC m=+76.976311815" watchObservedRunningTime="2026-07-17 21:46:53.587918134 +0000 UTC m=+76.976934785"
Jul 17 21:46:53.604139 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:53.604053    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hf7sc" podStartSLOduration=66.752997251 podStartE2EDuration="1m16.604030388s" totalImagesPullingTime="9.851033137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:53.602862996 +0000 UTC m=+76.991879648" watchObservedRunningTime="2026-07-17 21:46:53.604030388 +0000 UTC m=+76.993047040"
Jul 17 21:46:53.621441 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:53.621376    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64b4b5b48-wgpvl" podStartSLOduration=1.759310208 podStartE2EDuration="9.621354459s" totalImagesPullingTime="7.862044251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:53.620981318 +0000 UTC m=+77.009997973" watchObservedRunningTime="2026-07-17 21:46:53.621354459 +0000 UTC m=+77.010371110"
Jul 17 21:46:54.949373 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:54.949334    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:54.949828 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:54.949407    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:54.955939 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:54.955912    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:55.557959 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:55.557927    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:55.566357 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:55.566330    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:46:55.657308 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:55.657275    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:55.776276 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:55.776241    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:56.627953 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:56.627917    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:56.721947 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:56.721907    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:56.969955 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:56.969870    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:57.070883 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:57.070846    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:58.022156 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:58.021982    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64b4b5b48-wgpvl"]
Jul 17 21:46:58.772860 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:58.772821    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:58.872822 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:58.872783    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:58.982420 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:58.982387    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:59.075963 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:59.075881    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:46:59.356552 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:46:59.356476    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:00.286178 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:00.286143    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:00.387871 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:00.387840    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:00.943183 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:00.943146    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:01.044141 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:01.044102    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:01.585774 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:01.585739    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-8p6dp" event={"ID":"7929fd6d-36e0-46bf-83e6-7a93999d2f01","Type":"ContainerStarted","Data":"1527cacacf8681de6c1db9bb636404ffbbe23484edca699f749cabe48ccd7e40"}
Jul 17 21:47:01.586294 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:01.585967    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-8p6dp"
Jul 17 21:47:01.587662 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:01.587630    2565 patch_prober.go:28] interesting pod/downloads-5bc67dcd46-8p6dp container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.13:8080/\": dial tcp 10.133.0.13:8080: connect: connection refused" start-of-body=
Jul 17 21:47:01.587777 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:01.587692    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-5bc67dcd46-8p6dp" podUID="7929fd6d-36e0-46bf-83e6-7a93999d2f01" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.13:8080/\": dial tcp 10.133.0.13:8080: connect: connection refused"
Jul 17 21:47:01.607497 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:01.607445    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-8p6dp" podStartSLOduration=1.009704494 podStartE2EDuration="21.607429712s" totalImagesPullingTime="20.597725218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:01.606335674 +0000 UTC m=+84.995352326" watchObservedRunningTime="2026-07-17 21:47:01.607429712 +0000 UTC m=+84.996446363"
Jul 17 21:47:02.475749 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:02.475720    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55498b58b5-9kpmt"
Jul 17 21:47:02.502309 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:02.502267    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:02.601726 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:02.601695    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:02.605165 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:02.605139    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-8p6dp"
Jul 17 21:47:02.702241 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:02.702208    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:03.976824 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:03.976780    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:04.206419 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:04.206380    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:04.306824 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:04.306785    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:04.407655 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:04.407623    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:05.334233 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.334189    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54b875dd99-cs4hm" podUID="18e6a280-24ab-486f-9870-83b1c68cc38b" containerName="registry" containerID="cri-o://228a8a67d2c9829792889f63ab3498396e079ee53a3643f43dc31f41a0828765" gracePeriod=30
Jul 17 21:47:05.603722 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.603687    2565 generic.go:356] "Generic (PLEG): container finished" podID="18e6a280-24ab-486f-9870-83b1c68cc38b" containerID="228a8a67d2c9829792889f63ab3498396e079ee53a3643f43dc31f41a0828765" exitCode=0
Jul 17 21:47:05.603839 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.603760    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54b875dd99-cs4hm" event={"ID":"18e6a280-24ab-486f-9870-83b1c68cc38b","Type":"ContainerDied","Data":"228a8a67d2c9829792889f63ab3498396e079ee53a3643f43dc31f41a0828765"}
Jul 17 21:47:05.603839 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.603823    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54b875dd99-cs4hm" event={"ID":"18e6a280-24ab-486f-9870-83b1c68cc38b","Type":"ContainerDied","Data":"aa17292cfa598c58959b280a9fb3de3407e3ac51b3c9446e217c36498685687d"}
Jul 17 21:47:05.603931 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.603843    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa17292cfa598c58959b280a9fb3de3407e3ac51b3c9446e217c36498685687d"
Jul 17 21:47:05.629216 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.629181    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:47:05.768061 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.768024    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-bound-sa-token\") pod \"18e6a280-24ab-486f-9870-83b1c68cc38b\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") "
Jul 17 21:47:05.768326 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.768094    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18e6a280-24ab-486f-9870-83b1c68cc38b-trusted-ca\") pod \"18e6a280-24ab-486f-9870-83b1c68cc38b\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") "
Jul 17 21:47:05.768326 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.768128    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-certificates\") pod \"18e6a280-24ab-486f-9870-83b1c68cc38b\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") "
Jul 17 21:47:05.768326 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.768152    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls\") pod \"18e6a280-24ab-486f-9870-83b1c68cc38b\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") "
Jul 17 21:47:05.768326 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.768182    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mk4j\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-kube-api-access-9mk4j\") pod \"18e6a280-24ab-486f-9870-83b1c68cc38b\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") "
Jul 17 21:47:05.768326 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.768224    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18e6a280-24ab-486f-9870-83b1c68cc38b-image-registry-private-configuration\") pod \"18e6a280-24ab-486f-9870-83b1c68cc38b\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") "
Jul 17 21:47:05.768326 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.768259    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18e6a280-24ab-486f-9870-83b1c68cc38b-ca-trust-extracted\") pod \"18e6a280-24ab-486f-9870-83b1c68cc38b\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") "
Jul 17 21:47:05.768643 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.768338    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18e6a280-24ab-486f-9870-83b1c68cc38b-installation-pull-secrets\") pod \"18e6a280-24ab-486f-9870-83b1c68cc38b\" (UID: \"18e6a280-24ab-486f-9870-83b1c68cc38b\") "
Jul 17 21:47:05.768643 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.768518    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e6a280-24ab-486f-9870-83b1c68cc38b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "18e6a280-24ab-486f-9870-83b1c68cc38b" (UID: "18e6a280-24ab-486f-9870-83b1c68cc38b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:47:05.768643 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.768572    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "18e6a280-24ab-486f-9870-83b1c68cc38b" (UID: "18e6a280-24ab-486f-9870-83b1c68cc38b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:47:05.773603 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.773544    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e6a280-24ab-486f-9870-83b1c68cc38b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "18e6a280-24ab-486f-9870-83b1c68cc38b" (UID: "18e6a280-24ab-486f-9870-83b1c68cc38b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:47:05.773603 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.773592    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "18e6a280-24ab-486f-9870-83b1c68cc38b" (UID: "18e6a280-24ab-486f-9870-83b1c68cc38b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:47:05.773819 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.773772    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-kube-api-access-9mk4j" (OuterVolumeSpecName: "kube-api-access-9mk4j") pod "18e6a280-24ab-486f-9870-83b1c68cc38b" (UID: "18e6a280-24ab-486f-9870-83b1c68cc38b"). InnerVolumeSpecName "kube-api-access-9mk4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:47:05.776288 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.776252    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e6a280-24ab-486f-9870-83b1c68cc38b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "18e6a280-24ab-486f-9870-83b1c68cc38b" (UID: "18e6a280-24ab-486f-9870-83b1c68cc38b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:47:05.776368 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.776288    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "18e6a280-24ab-486f-9870-83b1c68cc38b" (UID: "18e6a280-24ab-486f-9870-83b1c68cc38b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:47:05.779517 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.779491    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e6a280-24ab-486f-9870-83b1c68cc38b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "18e6a280-24ab-486f-9870-83b1c68cc38b" (UID: "18e6a280-24ab-486f-9870-83b1c68cc38b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:47:05.869591 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.869506    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/18e6a280-24ab-486f-9870-83b1c68cc38b-installation-pull-secrets\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:47:05.869591 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.869548    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-bound-sa-token\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:47:05.869591 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.869565    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/18e6a280-24ab-486f-9870-83b1c68cc38b-trusted-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:47:05.869591 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.869578    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-certificates\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:47:05.869591 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.869587    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-registry-tls\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:47:05.869591 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.869597    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mk4j\" (UniqueName: \"kubernetes.io/projected/18e6a280-24ab-486f-9870-83b1c68cc38b-kube-api-access-9mk4j\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:47:05.869966 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.869626    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/18e6a280-24ab-486f-9870-83b1c68cc38b-image-registry-private-configuration\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:47:05.869966 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:05.869641    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/18e6a280-24ab-486f-9870-83b1c68cc38b-ca-trust-extracted\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:47:06.608087 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:06.608029    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54b875dd99-cs4hm"
Jul 17 21:47:06.637150 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:06.631939    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54b875dd99-cs4hm"]
Jul 17 21:47:06.637150 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:06.636708    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-54b875dd99-cs4hm"]
Jul 17 21:47:07.095883 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.095836    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:07.181895 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.181864    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18e6a280-24ab-486f-9870-83b1c68cc38b" path="/var/lib/kubelet/pods/18e6a280-24ab-486f-9870-83b1c68cc38b/volumes"
Jul 17 21:47:07.196054 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.196005    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:07.313006 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.312941    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:07.351739 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.351649    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zq9m4"]
Jul 17 21:47:07.352021 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.351997    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e6a280-24ab-486f-9870-83b1c68cc38b" containerName="registry"
Jul 17 21:47:07.352021 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.352021    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e6a280-24ab-486f-9870-83b1c68cc38b" containerName="registry"
Jul 17 21:47:07.352183 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.352114    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="18e6a280-24ab-486f-9870-83b1c68cc38b" containerName="registry"
Jul 17 21:47:07.383894 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.383864    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.386951 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.386922    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 21:47:07.387559 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.387539    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 21:47:07.387900 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.387882    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 21:47:07.388038 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.387918    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 21:47:07.388148 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.387929    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 21:47:07.388148 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.387958    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 21:47:07.388259 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.387977    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5ndnc\""
Jul 17 21:47:07.409492 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.409453    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:07.481506 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.481462    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-accelerators-collector-config\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.481678 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.481533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-wtmp\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.481678 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.481588    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f183d4-25b4-4f70-a9df-b82f914e7b58-sys\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.481678 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.481603    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf7fn\" (UniqueName: \"kubernetes.io/projected/48f183d4-25b4-4f70-a9df-b82f914e7b58-kube-api-access-hf7fn\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.481816 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.481676    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-tls\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.481816 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.481730    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f183d4-25b4-4f70-a9df-b82f914e7b58-metrics-client-ca\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.481816 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.481757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-textfile\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.481816 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.481787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48f183d4-25b4-4f70-a9df-b82f914e7b58-root\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.481985 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.481831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.521347 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.521313    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:07.583255 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.583221    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-tls\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.583444 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.583271    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f183d4-25b4-4f70-a9df-b82f914e7b58-metrics-client-ca\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.583444 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.583299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-textfile\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.583444 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.583332    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48f183d4-25b4-4f70-a9df-b82f914e7b58-root\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.583444 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.583364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.583444 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:47:07.583393    2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 17 21:47:07.583692 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.583441    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-accelerators-collector-config\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.583692 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:47:07.583475    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-tls podName:48f183d4-25b4-4f70-a9df-b82f914e7b58 nodeName:}" failed. No retries permitted until 2026-07-17 21:47:08.083451811 +0000 UTC m=+91.472468443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-tls") pod "node-exporter-zq9m4" (UID: "48f183d4-25b4-4f70-a9df-b82f914e7b58") : secret "node-exporter-tls" not found
Jul 17 21:47:07.583692 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.583528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-wtmp\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.583692 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.583572    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f183d4-25b4-4f70-a9df-b82f914e7b58-sys\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.583692 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.583595    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf7fn\" (UniqueName: \"kubernetes.io/projected/48f183d4-25b4-4f70-a9df-b82f914e7b58-kube-api-access-hf7fn\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.583940 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.583728    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-textfile\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.583940 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.583874    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-wtmp\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.583940 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.583905    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f183d4-25b4-4f70-a9df-b82f914e7b58-sys\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.583940 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.583930    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48f183d4-25b4-4f70-a9df-b82f914e7b58-root\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.584156 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.584009    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f183d4-25b4-4f70-a9df-b82f914e7b58-metrics-client-ca\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.584156 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.584030    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-accelerators-collector-config\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.586243 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.586222    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.593460 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.593430    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf7fn\" (UniqueName: \"kubernetes.io/projected/48f183d4-25b4-4f70-a9df-b82f914e7b58-kube-api-access-hf7fn\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:07.622938 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.622856    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:07.723541 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.723492    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:07.791297 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:07.791234    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:08.088907 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:08.088872    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-tls\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:08.091856 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:08.091825    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48f183d4-25b4-4f70-a9df-b82f914e7b58-node-exporter-tls\") pod \"node-exporter-zq9m4\" (UID: \"48f183d4-25b4-4f70-a9df-b82f914e7b58\") " pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:08.108367 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:08.108334    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:08.297953 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:08.297921    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zq9m4"
Jul 17 21:47:08.325625 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:08.325590    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:08.351027 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:47:08.350562    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f183d4_25b4_4f70_a9df_b82f914e7b58.slice/crio-f706072de0e9af6041d7e424fed5d2aa241197a47559625beabd69cd8bf8cf67 WatchSource:0}: Error finding container f706072de0e9af6041d7e424fed5d2aa241197a47559625beabd69cd8bf8cf67: Status 404 returned error can't find the container with id f706072de0e9af6041d7e424fed5d2aa241197a47559625beabd69cd8bf8cf67
Jul 17 21:47:08.406423 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:08.406382    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:08.618687 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:08.618600    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zq9m4" event={"ID":"48f183d4-25b4-4f70-a9df-b82f914e7b58","Type":"ContainerStarted","Data":"f706072de0e9af6041d7e424fed5d2aa241197a47559625beabd69cd8bf8cf67"}
Jul 17 21:47:08.630801 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:08.630754    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:08.727838 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:08.727803    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:08.828320 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:08.828284    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:10.311329 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:10.311258    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:10.414341 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:10.414227    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:10.631983 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:10.631943    2565 generic.go:356] "Generic (PLEG): container finished" podID="48f183d4-25b4-4f70-a9df-b82f914e7b58" containerID="248d531fc7e0a6491f30be9420093ab7b8e0d40eecc05ecdc51329b4e193775c" exitCode=0
Jul 17 21:47:10.632174 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:10.632031    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zq9m4" event={"ID":"48f183d4-25b4-4f70-a9df-b82f914e7b58","Type":"ContainerDied","Data":"248d531fc7e0a6491f30be9420093ab7b8e0d40eecc05ecdc51329b4e193775c"}
Jul 17 21:47:10.653184 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:10.653145    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:10.752112 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:10.752056    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:10.852208 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:10.852173    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:11.067290 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:11.067257    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:11.637632 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:11.637592    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zq9m4" event={"ID":"48f183d4-25b4-4f70-a9df-b82f914e7b58","Type":"ContainerStarted","Data":"d8e21ac5fc1381faa297e4bed076e9174403381643d42d247c406426564ecd3e"}
Jul 17 21:47:11.638038 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:11.637649    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zq9m4" event={"ID":"48f183d4-25b4-4f70-a9df-b82f914e7b58","Type":"ContainerStarted","Data":"f57ed56bdd8a0dc5aa20753a5c108a18c128e2ffc868f994253df36fc792e72c"}
Jul 17 21:47:11.658161 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:11.658100    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zq9m4" podStartSLOduration=2.7194944210000003 podStartE2EDuration="4.658045065s" totalImagesPullingTime="1.811021352s" totalInitContainerRuntime="127.529292ms" isStatefulPod=true podCreationTimestamp="2026-07-17 21:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:11.656850138 +0000 UTC m=+95.045866802" watchObservedRunningTime="2026-07-17 21:47:11.658045065 +0000 UTC m=+95.047061716"
Jul 17 21:47:12.157262 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:12.157231    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:12.249169 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:12.249136    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:12.347388 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:12.347350    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:12.450666 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:12.450594    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:12.547843 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:12.547807    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:13.588193 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.588161    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 21:47:13.589478 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.589443    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:13.671855 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.671819    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 21:47:13.672022 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.671917    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.674911 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.674887    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 17 21:47:13.675080 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.674921    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zrnj5\""
Jul 17 21:47:13.675080 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.674948    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 17 21:47:13.675080 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.674985    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 17 21:47:13.675080 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.675020    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 17 21:47:13.675253 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.675081    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 17 21:47:13.675485 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.675468    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 17 21:47:13.675547 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.675469    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 17 21:47:13.675600 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.675550    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 17 21:47:13.675600 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.675574    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 17 21:47:13.675696 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.675626    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 17 21:47:13.675953 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.675938    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 17 21:47:13.676013 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.675945    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 17 21:47:13.676742 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.676728    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 17 21:47:13.676814 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.676731    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-53cn7vtctgqm3\""
Jul 17 21:47:13.679382 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.679365    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 17 21:47:13.681107 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.681091    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 17 21:47:13.836657 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.836626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.836657 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.836662    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.836914 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.836685    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.836914 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.836720    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.836914 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.836755    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-config-out\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.836914 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.836769    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.836914 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.836793    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.836914 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.836844    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.836914 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.836890    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.837299 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.836927    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.837299 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.836968    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.837299 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.836992    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.837299 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.837039    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.837299 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.837118    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-config\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.837299 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.837143    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.837299 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.837172    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.837299 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.837201    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.837299 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.837226    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.837299 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.837262    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-web-config\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.837299 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.837288    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79rp2\" (UniqueName: \"kubernetes.io/projected/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-kube-api-access-79rp2\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938093 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.937985    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938093 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.938025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938093 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.938046    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938321 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.938191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-config\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938321 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.938223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938321 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.938248    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938321 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.938277    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938612 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.938588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938694 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.938638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-web-config\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938694 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.938669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79rp2\" (UniqueName: \"kubernetes.io/projected/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-kube-api-access-79rp2\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938805 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.938715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938805 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.938748    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938805 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.938787    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938949 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.938814    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938949 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.938822    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938949 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.938641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.938949 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.938877    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.939525 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.939496    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.939912 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.939889    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-config-out\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.939996 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.939965    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.940060 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.940021    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.940060 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.940048    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.940199 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.940092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.940199 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.940118    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.940199 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.940122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.940858 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.940837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.941519 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.941492    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.941764 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.941738    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.941833 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.941750    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.941833 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.941804    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-config\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.941933 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.941843    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.942877 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.942563    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.943214 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.943189    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.943818 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.943772    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-config-out\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.943914 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.943845    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.944217 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.944195    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.944501 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.944484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.944652 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.944630    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.945329 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.945306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-web-config\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.946882 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.946865    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79rp2\" (UniqueName: \"kubernetes.io/projected/d9c621c2-d1a4-4b88-83c4-6d6aae5750b6-kube-api-access-79rp2\") pod \"prometheus-k8s-0\" (UID: \"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:13.982660 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:13.982617    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:14.150199 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:14.150113    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 21:47:14.152868 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:47:14.152838    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c621c2_d1a4_4b88_83c4_6d6aae5750b6.slice/crio-90110947d8e7be486d489dd2c929d4a56135e795184925d545203ea9c7f99a31 WatchSource:0}: Error finding container 90110947d8e7be486d489dd2c929d4a56135e795184925d545203ea9c7f99a31: Status 404 returned error can't find the container with id 90110947d8e7be486d489dd2c929d4a56135e795184925d545203ea9c7f99a31
Jul 17 21:47:14.649803 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:14.649768    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6","Type":"ContainerStarted","Data":"90110947d8e7be486d489dd2c929d4a56135e795184925d545203ea9c7f99a31"}
Jul 17 21:47:16.660296 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:16.660253    2565 generic.go:356] "Generic (PLEG): container finished" podID="d9c621c2-d1a4-4b88-83c4-6d6aae5750b6" containerID="8f6a63b3f6dfaf8fc49f5c459256f6ba39897040bb4fe950d3cbbdfb29b09cbf" exitCode=0
Jul 17 21:47:16.660870 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:16.660342    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6","Type":"ContainerDied","Data":"8f6a63b3f6dfaf8fc49f5c459256f6ba39897040bb4fe950d3cbbdfb29b09cbf"}
Jul 17 21:47:17.581666 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:17.581609    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:20.678021 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:20.677986    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6","Type":"ContainerStarted","Data":"062fb04c8f599de14847df3d37fb388033d6933542c905a5eb777263b270dc34"}
Jul 17 21:47:20.678464 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:20.678052    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6","Type":"ContainerStarted","Data":"a101debe9faa96bb93b166beecb7ca6f519e740b61ed4d0cb27098af74db4151"}
Jul 17 21:47:23.285906 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:23.285858    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:23.337893 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:23.337862    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:23.395336 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:23.395299    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:23.604411 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:23.604320    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64b4b5b48-wgpvl" podUID="8658b73d-a8e3-4934-bca6-26e294bf7000" containerName="console" containerID="cri-o://b60eb1dad5771b897e8a2a43a3df7023762c809861d08418c28ffe8209d6dd87" gracePeriod=15
Jul 17 21:47:23.691944 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:23.691908    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6","Type":"ContainerStarted","Data":"bd84423bd82f4a3d49a1944b6d16f24490bbfb8d3faaac8a49ec90ca72592578"}
Jul 17 21:47:23.692119 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:23.691988    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6","Type":"ContainerStarted","Data":"288ff8195c62cc2cac35733f99b1573fd054a718c292de71e07fd4a772738e68"}
Jul 17 21:47:23.692119 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:23.692028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6","Type":"ContainerStarted","Data":"47d2af160227116fc10e49031a1fb184b049231857de95cb7e0fa5cebd9b5774"}
Jul 17 21:47:23.692119 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:23.692081    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9c621c2-d1a4-4b88-83c4-6d6aae5750b6","Type":"ContainerStarted","Data":"bbd20b74ff5e956404b0225de984686e2617baac28d2ebd47815cd981dec4b0e"}
Jul 17 21:47:23.725860 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:23.725783    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.486697386 podStartE2EDuration="10.725738261s" totalImagesPullingTime="7.226662783s" totalInitContainerRuntime="12.378092ms" isStatefulPod=false podCreationTimestamp="2026-07-17 21:47:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:23.720957232 +0000 UTC m=+107.109973882" watchObservedRunningTime="2026-07-17 21:47:23.725738261 +0000 UTC m=+107.114754913"
Jul 17 21:47:23.887688 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:23.887656    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:47:23.983201 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:23.983150    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:47:24.029667 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.029572    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8658b73d-a8e3-4934-bca6-26e294bf7000-console-oauth-config\") pod \"8658b73d-a8e3-4934-bca6-26e294bf7000\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") "
Jul 17 21:47:24.029780 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.029712    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-oauth-serving-cert\") pod \"8658b73d-a8e3-4934-bca6-26e294bf7000\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") "
Jul 17 21:47:24.029780 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.029733    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-service-ca\") pod \"8658b73d-a8e3-4934-bca6-26e294bf7000\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") "
Jul 17 21:47:24.029780 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.029751    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-console-config\") pod \"8658b73d-a8e3-4934-bca6-26e294bf7000\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") "
Jul 17 21:47:24.029780 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.029773    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltp92\" (UniqueName: \"kubernetes.io/projected/8658b73d-a8e3-4934-bca6-26e294bf7000-kube-api-access-ltp92\") pod \"8658b73d-a8e3-4934-bca6-26e294bf7000\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") "
Jul 17 21:47:24.029979 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.029807    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8658b73d-a8e3-4934-bca6-26e294bf7000-console-serving-cert\") pod \"8658b73d-a8e3-4934-bca6-26e294bf7000\" (UID: \"8658b73d-a8e3-4934-bca6-26e294bf7000\") "
Jul 17 21:47:24.030180 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.030150    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8658b73d-a8e3-4934-bca6-26e294bf7000" (UID: "8658b73d-a8e3-4934-bca6-26e294bf7000"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:47:24.030320 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.030182    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-console-config" (OuterVolumeSpecName: "console-config") pod "8658b73d-a8e3-4934-bca6-26e294bf7000" (UID: "8658b73d-a8e3-4934-bca6-26e294bf7000"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:47:24.030320 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.030182    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-service-ca" (OuterVolumeSpecName: "service-ca") pod "8658b73d-a8e3-4934-bca6-26e294bf7000" (UID: "8658b73d-a8e3-4934-bca6-26e294bf7000"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:47:24.032179 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.032147    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8658b73d-a8e3-4934-bca6-26e294bf7000-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8658b73d-a8e3-4934-bca6-26e294bf7000" (UID: "8658b73d-a8e3-4934-bca6-26e294bf7000"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:47:24.032284 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.032199    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8658b73d-a8e3-4934-bca6-26e294bf7000-kube-api-access-ltp92" (OuterVolumeSpecName: "kube-api-access-ltp92") pod "8658b73d-a8e3-4934-bca6-26e294bf7000" (UID: "8658b73d-a8e3-4934-bca6-26e294bf7000"). InnerVolumeSpecName "kube-api-access-ltp92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:47:24.032284 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.032254    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8658b73d-a8e3-4934-bca6-26e294bf7000-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8658b73d-a8e3-4934-bca6-26e294bf7000" (UID: "8658b73d-a8e3-4934-bca6-26e294bf7000"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:47:24.130763 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.130676    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8658b73d-a8e3-4934-bca6-26e294bf7000-console-oauth-config\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:47:24.130763 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.130707    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-oauth-serving-cert\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:47:24.130763 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.130716    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:47:24.130763 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.130726    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8658b73d-a8e3-4934-bca6-26e294bf7000-console-config\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:47:24.130763 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.130735    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltp92\" (UniqueName: \"kubernetes.io/projected/8658b73d-a8e3-4934-bca6-26e294bf7000-kube-api-access-ltp92\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:47:24.130763 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.130743    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8658b73d-a8e3-4934-bca6-26e294bf7000-console-serving-cert\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:47:24.560861 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.560834    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mg8bv"
Jul 17 21:47:24.697602 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.697567    2565 generic.go:356] "Generic (PLEG): container finished" podID="8658b73d-a8e3-4934-bca6-26e294bf7000" containerID="b60eb1dad5771b897e8a2a43a3df7023762c809861d08418c28ffe8209d6dd87" exitCode=0
Jul 17 21:47:24.697872 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.697610    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b4b5b48-wgpvl"
Jul 17 21:47:24.697872 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.697639    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b4b5b48-wgpvl" event={"ID":"8658b73d-a8e3-4934-bca6-26e294bf7000","Type":"ContainerDied","Data":"b60eb1dad5771b897e8a2a43a3df7023762c809861d08418c28ffe8209d6dd87"}
Jul 17 21:47:24.697872 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.697679    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b4b5b48-wgpvl" event={"ID":"8658b73d-a8e3-4934-bca6-26e294bf7000","Type":"ContainerDied","Data":"16b34ecd58e01e6c4df108db1f2ffc6dccd1ea7084850f6bce24f8a7a0b49a4f"}
Jul 17 21:47:24.697872 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.697698    2565 scope.go:117] "RemoveContainer" containerID="b60eb1dad5771b897e8a2a43a3df7023762c809861d08418c28ffe8209d6dd87"
Jul 17 21:47:24.707204 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.707183    2565 scope.go:117] "RemoveContainer" containerID="b60eb1dad5771b897e8a2a43a3df7023762c809861d08418c28ffe8209d6dd87"
Jul 17 21:47:24.707505 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:47:24.707485    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60eb1dad5771b897e8a2a43a3df7023762c809861d08418c28ffe8209d6dd87\": container with ID starting with b60eb1dad5771b897e8a2a43a3df7023762c809861d08418c28ffe8209d6dd87 not found: ID does not exist" containerID="b60eb1dad5771b897e8a2a43a3df7023762c809861d08418c28ffe8209d6dd87"
Jul 17 21:47:24.707568 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.707514    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60eb1dad5771b897e8a2a43a3df7023762c809861d08418c28ffe8209d6dd87"} err="failed to get container status \"b60eb1dad5771b897e8a2a43a3df7023762c809861d08418c28ffe8209d6dd87\": rpc error: code = NotFound desc = could not find container \"b60eb1dad5771b897e8a2a43a3df7023762c809861d08418c28ffe8209d6dd87\": container with ID starting with b60eb1dad5771b897e8a2a43a3df7023762c809861d08418c28ffe8209d6dd87 not found: ID does not exist"
Jul 17 21:47:24.717626 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.717603    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64b4b5b48-wgpvl"]
Jul 17 21:47:24.721271 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:24.721248    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64b4b5b48-wgpvl"]
Jul 17 21:47:25.181237 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:25.181201    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8658b73d-a8e3-4934-bca6-26e294bf7000" path="/var/lib/kubelet/pods/8658b73d-a8e3-4934-bca6-26e294bf7000/volumes"
Jul 17 21:47:25.727402 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:25.727369    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:25.827368 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:25.827330    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:40.676941 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:40.676910    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:40.732727 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:40.732697    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:44.854471 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:44.854436    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:47:44.907268 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:47:44.907232    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:48:13.983314 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:13.983269    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:48:14.002346 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:14.002320    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:48:14.897593 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:14.897558    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 21:48:26.591858 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:26.591824    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:48:26.706540 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:26.706499    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:48:35.141901 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:35.141803    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:48:35.243331 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:35.243298    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:48:35.343767 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:35.343738    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:48:35.872983 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:35.872942    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:48:35.973629 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:35.973600    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:48:41.828644 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:41.828615    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:48:41.881863 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:41.881835    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:48:53.753752 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:53.753711    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:48:53.854990 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:53.854956    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:48:54.218131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:54.218021    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:48:56.691996 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:56.691958    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:48:56.792857 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:56.792826    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:48:56.841239 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:48:56.841203    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:10.661461 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:10.661426    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:10.762017 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:10.761982    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:12.586442 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:12.586410    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:12.684043 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:12.684011    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:26.531885 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:26.531854    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:26.631144 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:26.631114    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:29.758078 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:29.758020    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:29.849237 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:29.849203    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:29.914970 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:29.914933    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:34.114516 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:34.114480    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:50.048932 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:50.048900    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:50.103782 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:50.103752    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:53.266299 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:53.266274    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:53.394123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:53.394090    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:54.106606 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:54.106571    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:57.077572 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:57.077535    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:49:57.204742 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:49:57.204703    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:01.908530 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:01.908452    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:01.961562 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:01.961527    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:08.747797 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:08.747767    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:08.855975 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:08.855938    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:09.313773 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:09.313736    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:09.436324 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:09.436289    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:10.965739 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:10.965705    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:11.156665 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:11.156632    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:15.796809 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:15.796766    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:15.864047 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:15.864014    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:18.959495 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:18.959461    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:19.248240 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.248209    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-28r7n"]
Jul 17 21:50:19.248483 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.248470    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8658b73d-a8e3-4934-bca6-26e294bf7000" containerName="console"
Jul 17 21:50:19.248525 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.248484    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8658b73d-a8e3-4934-bca6-26e294bf7000" containerName="console"
Jul 17 21:50:19.248562 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.248531    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8658b73d-a8e3-4934-bca6-26e294bf7000" containerName="console"
Jul 17 21:50:19.251528 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.251513    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-28r7n"
Jul 17 21:50:19.254172 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.254144    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 17 21:50:19.254304 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.254217    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 17 21:50:19.255307 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.255287    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-cbzt6\""
Jul 17 21:50:19.261558 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.261534    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-28r7n"]
Jul 17 21:50:19.401716 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.401676    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zkdg\" (UniqueName: \"kubernetes.io/projected/ce82d864-f26a-4bfa-bf74-48b97509eb22-kube-api-access-9zkdg\") pod \"cert-manager-55b4d954d5-28r7n\" (UID: \"ce82d864-f26a-4bfa-bf74-48b97509eb22\") " pod="cert-manager/cert-manager-55b4d954d5-28r7n"
Jul 17 21:50:19.401899 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.401749    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce82d864-f26a-4bfa-bf74-48b97509eb22-bound-sa-token\") pod \"cert-manager-55b4d954d5-28r7n\" (UID: \"ce82d864-f26a-4bfa-bf74-48b97509eb22\") " pod="cert-manager/cert-manager-55b4d954d5-28r7n"
Jul 17 21:50:19.502703 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.502612    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce82d864-f26a-4bfa-bf74-48b97509eb22-bound-sa-token\") pod \"cert-manager-55b4d954d5-28r7n\" (UID: \"ce82d864-f26a-4bfa-bf74-48b97509eb22\") " pod="cert-manager/cert-manager-55b4d954d5-28r7n"
Jul 17 21:50:19.502703 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.502691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zkdg\" (UniqueName: \"kubernetes.io/projected/ce82d864-f26a-4bfa-bf74-48b97509eb22-kube-api-access-9zkdg\") pod \"cert-manager-55b4d954d5-28r7n\" (UID: \"ce82d864-f26a-4bfa-bf74-48b97509eb22\") " pod="cert-manager/cert-manager-55b4d954d5-28r7n"
Jul 17 21:50:19.510882 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.510853    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce82d864-f26a-4bfa-bf74-48b97509eb22-bound-sa-token\") pod \"cert-manager-55b4d954d5-28r7n\" (UID: \"ce82d864-f26a-4bfa-bf74-48b97509eb22\") " pod="cert-manager/cert-manager-55b4d954d5-28r7n"
Jul 17 21:50:19.510970 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.510865    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zkdg\" (UniqueName: \"kubernetes.io/projected/ce82d864-f26a-4bfa-bf74-48b97509eb22-kube-api-access-9zkdg\") pod \"cert-manager-55b4d954d5-28r7n\" (UID: \"ce82d864-f26a-4bfa-bf74-48b97509eb22\") " pod="cert-manager/cert-manager-55b4d954d5-28r7n"
Jul 17 21:50:19.567999 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.567949    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-28r7n"
Jul 17 21:50:19.703603 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:19.703571    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-28r7n"]
Jul 17 21:50:19.707297 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:50:19.707271    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce82d864_f26a_4bfa_bf74_48b97509eb22.slice/crio-81eed2a5aa73adcb231df0a84d5eaa7a40170d3ea46d412584e9c444d58079cc WatchSource:0}: Error finding container 81eed2a5aa73adcb231df0a84d5eaa7a40170d3ea46d412584e9c444d58079cc: Status 404 returned error can't find the container with id 81eed2a5aa73adcb231df0a84d5eaa7a40170d3ea46d412584e9c444d58079cc
Jul 17 21:50:20.319345 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:20.319304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-28r7n" event={"ID":"ce82d864-f26a-4bfa-bf74-48b97509eb22","Type":"ContainerStarted","Data":"81eed2a5aa73adcb231df0a84d5eaa7a40170d3ea46d412584e9c444d58079cc"}
Jul 17 21:50:21.216554 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:21.216512    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:21.314407 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:21.314368    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:24.336848 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:24.336803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-28r7n" event={"ID":"ce82d864-f26a-4bfa-bf74-48b97509eb22","Type":"ContainerStarted","Data":"23b9876422d07be390cac2e685ee048afaf343c1864302a318f1105c59fe452a"}
Jul 17 21:50:24.353391 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:24.353342    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-28r7n" podStartSLOduration=1.365924717 podStartE2EDuration="5.353323622s" totalImagesPullingTime="3.987398905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:50:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:50:24.351282077 +0000 UTC m=+287.740298722" watchObservedRunningTime="2026-07-17 21:50:24.353323622 +0000 UTC m=+287.742340273"
Jul 17 21:50:26.910752 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:26.910716    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:27.010508 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:27.010472    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:29.272588 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:29.272555    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:37.038029 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:37.037985    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 21:50:37.043297 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:37.043275    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 21:50:37.049717 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:37.049697    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 21:50:44.764551 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:44.764514    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:46.018372 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.014720    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"]
Jul 17 21:50:46.019017 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.018996    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.026297 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.026278    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 21:50:46.026576 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.026561    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 21:50:46.028120 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.028061    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 21:50:46.028326 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.028310    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rl47g\""
Jul 17 21:50:46.028577 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.028561    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 21:50:46.043416 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.043389    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"]
Jul 17 21:50:46.108582 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.108542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eec8dce4-acea-42e6-add7-879b08ce7aa8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bff9999cf-xpc64\" (UID: \"eec8dce4-acea-42e6-add7-879b08ce7aa8\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.108582 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.108596    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq825\" (UniqueName: \"kubernetes.io/projected/eec8dce4-acea-42e6-add7-879b08ce7aa8-kube-api-access-mq825\") pod \"opendatahub-operator-controller-manager-bff9999cf-xpc64\" (UID: \"eec8dce4-acea-42e6-add7-879b08ce7aa8\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.108828 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.108672    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eec8dce4-acea-42e6-add7-879b08ce7aa8-webhook-cert\") pod \"opendatahub-operator-controller-manager-bff9999cf-xpc64\" (UID: \"eec8dce4-acea-42e6-add7-879b08ce7aa8\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.108828 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.108731    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eec8dce4-acea-42e6-add7-879b08ce7aa8-manifests\") pod \"opendatahub-operator-controller-manager-bff9999cf-xpc64\" (UID: \"eec8dce4-acea-42e6-add7-879b08ce7aa8\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.108828 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.108780    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eec8dce4-acea-42e6-add7-879b08ce7aa8-tmp\") pod \"opendatahub-operator-controller-manager-bff9999cf-xpc64\" (UID: \"eec8dce4-acea-42e6-add7-879b08ce7aa8\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.209664 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.209625    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eec8dce4-acea-42e6-add7-879b08ce7aa8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bff9999cf-xpc64\" (UID: \"eec8dce4-acea-42e6-add7-879b08ce7aa8\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.209664 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.209670    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq825\" (UniqueName: \"kubernetes.io/projected/eec8dce4-acea-42e6-add7-879b08ce7aa8-kube-api-access-mq825\") pod \"opendatahub-operator-controller-manager-bff9999cf-xpc64\" (UID: \"eec8dce4-acea-42e6-add7-879b08ce7aa8\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.209913 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.209697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eec8dce4-acea-42e6-add7-879b08ce7aa8-webhook-cert\") pod \"opendatahub-operator-controller-manager-bff9999cf-xpc64\" (UID: \"eec8dce4-acea-42e6-add7-879b08ce7aa8\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.209913 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.209715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eec8dce4-acea-42e6-add7-879b08ce7aa8-manifests\") pod \"opendatahub-operator-controller-manager-bff9999cf-xpc64\" (UID: \"eec8dce4-acea-42e6-add7-879b08ce7aa8\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.209913 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.209744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eec8dce4-acea-42e6-add7-879b08ce7aa8-tmp\") pod \"opendatahub-operator-controller-manager-bff9999cf-xpc64\" (UID: \"eec8dce4-acea-42e6-add7-879b08ce7aa8\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.210274 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.210250    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eec8dce4-acea-42e6-add7-879b08ce7aa8-manifests\") pod \"opendatahub-operator-controller-manager-bff9999cf-xpc64\" (UID: \"eec8dce4-acea-42e6-add7-879b08ce7aa8\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.212259 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.212236    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eec8dce4-acea-42e6-add7-879b08ce7aa8-tmp\") pod \"opendatahub-operator-controller-manager-bff9999cf-xpc64\" (UID: \"eec8dce4-acea-42e6-add7-879b08ce7aa8\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.212357 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.212325    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eec8dce4-acea-42e6-add7-879b08ce7aa8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bff9999cf-xpc64\" (UID: \"eec8dce4-acea-42e6-add7-879b08ce7aa8\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.212464 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.212445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eec8dce4-acea-42e6-add7-879b08ce7aa8-webhook-cert\") pod \"opendatahub-operator-controller-manager-bff9999cf-xpc64\" (UID: \"eec8dce4-acea-42e6-add7-879b08ce7aa8\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.242022 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.241989    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq825\" (UniqueName: \"kubernetes.io/projected/eec8dce4-acea-42e6-add7-879b08ce7aa8-kube-api-access-mq825\") pod \"opendatahub-operator-controller-manager-bff9999cf-xpc64\" (UID: \"eec8dce4-acea-42e6-add7-879b08ce7aa8\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.329541 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.329439    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:46.387639 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.387552    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:46.472917 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.472887    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"]
Jul 17 21:50:46.476345 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:50:46.476312    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec8dce4_acea_42e6_add7_879b08ce7aa8.slice/crio-f59a8c66d82b69544952ea8817e61a94d8ad65ad2852445f0b6a040c1e6b1995 WatchSource:0}: Error finding container f59a8c66d82b69544952ea8817e61a94d8ad65ad2852445f0b6a040c1e6b1995: Status 404 returned error can't find the container with id f59a8c66d82b69544952ea8817e61a94d8ad65ad2852445f0b6a040c1e6b1995
Jul 17 21:50:46.478128 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.478111    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:50:46.504274 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.504240    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:46.623239 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.623134    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:46.806594 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.806560    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:46.943347 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:46.943235    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:47.117236 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:47.117199    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:47.254480 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:47.254445    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:47.411032 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:47.410997    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:47.430811 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:47.430669    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64" event={"ID":"eec8dce4-acea-42e6-add7-879b08ce7aa8","Type":"ContainerStarted","Data":"f59a8c66d82b69544952ea8817e61a94d8ad65ad2852445f0b6a040c1e6b1995"}
Jul 17 21:50:47.520499 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:47.520371    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:47.812371 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:47.812291    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:47.942794 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:47.942722    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:50.443723 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:50.443686    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64" event={"ID":"eec8dce4-acea-42e6-add7-879b08ce7aa8","Type":"ContainerStarted","Data":"8beb253fc039ed465bbf953f270850fba2367efbd306ce1d3b3c892e4c35b0e5"}
Jul 17 21:50:51.448560 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:51.448520    2565 generic.go:356] "Generic (PLEG): container finished" podID="eec8dce4-acea-42e6-add7-879b08ce7aa8" containerID="8beb253fc039ed465bbf953f270850fba2367efbd306ce1d3b3c892e4c35b0e5" exitCode=0
Jul 17 21:50:51.448934 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:51.448576    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64" event={"ID":"eec8dce4-acea-42e6-add7-879b08ce7aa8","Type":"ContainerDied","Data":"8beb253fc039ed465bbf953f270850fba2367efbd306ce1d3b3c892e4c35b0e5"}
Jul 17 21:50:52.454539 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:52.454500    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64" event={"ID":"eec8dce4-acea-42e6-add7-879b08ce7aa8","Type":"ContainerStarted","Data":"90beee73b69084c2fb04522924e4b46dfccc50914737c259f66f08e20321ebdc"}
Jul 17 21:50:52.455017 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:52.454613    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:50:52.476775 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:52.476715    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64" podStartSLOduration=2.917953086 podStartE2EDuration="7.476698484s" totalImagesPullingTime="3.475685082s" totalInitContainerRuntime="1.083060316s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:50:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:50:52.474920346 +0000 UTC m=+315.863936997" watchObservedRunningTime="2026-07-17 21:50:52.476698484 +0000 UTC m=+315.865715134"
Jul 17 21:50:57.594643 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:57.594610    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:50:57.697417 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:50:57.697379    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:03.459829 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:03.459797    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-xpc64"
Jul 17 21:51:05.854893 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:05.854858    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:05.996201 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:05.996163    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:06.322375 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.322339    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:06.481763 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.481719    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr"]
Jul 17 21:51:06.485141 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.485117    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr"
Jul 17 21:51:06.487831 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.487802    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 17 21:51:06.488754 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.488733    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 17 21:51:06.488754 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.488733    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 21:51:06.488908 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.488735    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 21:51:06.514693 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.514665    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:06.523914 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.523885    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr"]
Jul 17 21:51:06.584618 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.584531    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81716365-9c9f-4819-b6d4-e948453cb094-tls-certs\") pod \"kube-auth-proxy-85fdd95b7f-x9zvr\" (UID: \"81716365-9c9f-4819-b6d4-e948453cb094\") " pod="openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr"
Jul 17 21:51:06.584618 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.584570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81716365-9c9f-4819-b6d4-e948453cb094-tmp\") pod \"kube-auth-proxy-85fdd95b7f-x9zvr\" (UID: \"81716365-9c9f-4819-b6d4-e948453cb094\") " pod="openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr"
Jul 17 21:51:06.584797 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.584645    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42m8s\" (UniqueName: \"kubernetes.io/projected/81716365-9c9f-4819-b6d4-e948453cb094-kube-api-access-42m8s\") pod \"kube-auth-proxy-85fdd95b7f-x9zvr\" (UID: \"81716365-9c9f-4819-b6d4-e948453cb094\") " pod="openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr"
Jul 17 21:51:06.686052 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.685994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42m8s\" (UniqueName: \"kubernetes.io/projected/81716365-9c9f-4819-b6d4-e948453cb094-kube-api-access-42m8s\") pod \"kube-auth-proxy-85fdd95b7f-x9zvr\" (UID: \"81716365-9c9f-4819-b6d4-e948453cb094\") " pod="openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr"
Jul 17 21:51:06.686257 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.686100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81716365-9c9f-4819-b6d4-e948453cb094-tls-certs\") pod \"kube-auth-proxy-85fdd95b7f-x9zvr\" (UID: \"81716365-9c9f-4819-b6d4-e948453cb094\") " pod="openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr"
Jul 17 21:51:06.686257 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.686127    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81716365-9c9f-4819-b6d4-e948453cb094-tmp\") pod \"kube-auth-proxy-85fdd95b7f-x9zvr\" (UID: \"81716365-9c9f-4819-b6d4-e948453cb094\") " pod="openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr"
Jul 17 21:51:06.688379 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.688350    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81716365-9c9f-4819-b6d4-e948453cb094-tmp\") pod \"kube-auth-proxy-85fdd95b7f-x9zvr\" (UID: \"81716365-9c9f-4819-b6d4-e948453cb094\") " pod="openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr"
Jul 17 21:51:06.688791 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.688769    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81716365-9c9f-4819-b6d4-e948453cb094-tls-certs\") pod \"kube-auth-proxy-85fdd95b7f-x9zvr\" (UID: \"81716365-9c9f-4819-b6d4-e948453cb094\") " pod="openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr"
Jul 17 21:51:06.698108 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.698082    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42m8s\" (UniqueName: \"kubernetes.io/projected/81716365-9c9f-4819-b6d4-e948453cb094-kube-api-access-42m8s\") pod \"kube-auth-proxy-85fdd95b7f-x9zvr\" (UID: \"81716365-9c9f-4819-b6d4-e948453cb094\") " pod="openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr"
Jul 17 21:51:06.794196 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.794158    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr"
Jul 17 21:51:06.925017 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.924987    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z8r8m"]
Jul 17 21:51:06.928698 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.928675    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-z8r8m"
Jul 17 21:51:06.931682 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.931658    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 17 21:51:06.932327 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.932304    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-j9ttt\""
Jul 17 21:51:06.979848 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.979817    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr"]
Jul 17 21:51:06.982690 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:51:06.982665    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81716365_9c9f_4819_b6d4_e948453cb094.slice/crio-029056518e22d854d6a4ece919f1e91b4e448aba7a8ba3761f3f8e8097efb2ce WatchSource:0}: Error finding container 029056518e22d854d6a4ece919f1e91b4e448aba7a8ba3761f3f8e8097efb2ce: Status 404 returned error can't find the container with id 029056518e22d854d6a4ece919f1e91b4e448aba7a8ba3761f3f8e8097efb2ce
Jul 17 21:51:06.989132 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.989099    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/877e02a7-c53c-4331-b7d0-409aa961ec39-cert\") pod \"odh-model-controller-7998c67dc4-z8r8m\" (UID: \"877e02a7-c53c-4331-b7d0-409aa961ec39\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8r8m"
Jul 17 21:51:06.989223 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.989160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh45k\" (UniqueName: \"kubernetes.io/projected/877e02a7-c53c-4331-b7d0-409aa961ec39-kube-api-access-kh45k\") pod \"odh-model-controller-7998c67dc4-z8r8m\" (UID: \"877e02a7-c53c-4331-b7d0-409aa961ec39\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8r8m"
Jul 17 21:51:06.991180 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:06.991148    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:07.027812 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:07.027782    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z8r8m"]
Jul 17 21:51:07.090488 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:07.090452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/877e02a7-c53c-4331-b7d0-409aa961ec39-cert\") pod \"odh-model-controller-7998c67dc4-z8r8m\" (UID: \"877e02a7-c53c-4331-b7d0-409aa961ec39\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8r8m"
Jul 17 21:51:07.090691 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:07.090509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh45k\" (UniqueName: \"kubernetes.io/projected/877e02a7-c53c-4331-b7d0-409aa961ec39-kube-api-access-kh45k\") pod \"odh-model-controller-7998c67dc4-z8r8m\" (UID: \"877e02a7-c53c-4331-b7d0-409aa961ec39\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8r8m"
Jul 17 21:51:07.090691 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:51:07.090590    2565 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 17 21:51:07.090691 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:51:07.090668    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/877e02a7-c53c-4331-b7d0-409aa961ec39-cert podName:877e02a7-c53c-4331-b7d0-409aa961ec39 nodeName:}" failed. No retries permitted until 2026-07-17 21:51:07.590647042 +0000 UTC m=+330.979663672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/877e02a7-c53c-4331-b7d0-409aa961ec39-cert") pod "odh-model-controller-7998c67dc4-z8r8m" (UID: "877e02a7-c53c-4331-b7d0-409aa961ec39") : secret "odh-model-controller-webhook-cert" not found
Jul 17 21:51:07.101988 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:07.101923    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh45k\" (UniqueName: \"kubernetes.io/projected/877e02a7-c53c-4331-b7d0-409aa961ec39-kube-api-access-kh45k\") pod \"odh-model-controller-7998c67dc4-z8r8m\" (UID: \"877e02a7-c53c-4331-b7d0-409aa961ec39\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8r8m"
Jul 17 21:51:07.102522 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:07.102500    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:07.225505 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:07.225470    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:07.348621 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:07.348590    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:07.511553 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:07.511519    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr" event={"ID":"81716365-9c9f-4819-b6d4-e948453cb094","Type":"ContainerStarted","Data":"029056518e22d854d6a4ece919f1e91b4e448aba7a8ba3761f3f8e8097efb2ce"}
Jul 17 21:51:07.595135 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:07.595099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/877e02a7-c53c-4331-b7d0-409aa961ec39-cert\") pod \"odh-model-controller-7998c67dc4-z8r8m\" (UID: \"877e02a7-c53c-4331-b7d0-409aa961ec39\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8r8m"
Jul 17 21:51:07.598040 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:07.598013    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/877e02a7-c53c-4331-b7d0-409aa961ec39-cert\") pod \"odh-model-controller-7998c67dc4-z8r8m\" (UID: \"877e02a7-c53c-4331-b7d0-409aa961ec39\") " pod="opendatahub/odh-model-controller-7998c67dc4-z8r8m"
Jul 17 21:51:07.840325 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:07.840234    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-z8r8m"
Jul 17 21:51:07.990013 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:07.989959    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:08.012443 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:08.012414    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z8r8m"]
Jul 17 21:51:08.014641 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:51:08.014616    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877e02a7_c53c_4331_b7d0_409aa961ec39.slice/crio-1bd261116b966b257ae1e99dc1d42c7dc8d8be13a835f1870d2464afe8f1471c WatchSource:0}: Error finding container 1bd261116b966b257ae1e99dc1d42c7dc8d8be13a835f1870d2464afe8f1471c: Status 404 returned error can't find the container with id 1bd261116b966b257ae1e99dc1d42c7dc8d8be13a835f1870d2464afe8f1471c
Jul 17 21:51:08.167881 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:08.167799    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:08.517947 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:08.517912    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-z8r8m" event={"ID":"877e02a7-c53c-4331-b7d0-409aa961ec39","Type":"ContainerStarted","Data":"1bd261116b966b257ae1e99dc1d42c7dc8d8be13a835f1870d2464afe8f1471c"}
Jul 17 21:51:08.726433 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:08.726394    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:08.834004 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:08.833916    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:09.451139 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:09.451099    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:09.551214 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:09.551177    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:10.080013 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:10.079978    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:11.534935 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:11.534901    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-z8r8m" event={"ID":"877e02a7-c53c-4331-b7d0-409aa961ec39","Type":"ContainerStarted","Data":"e73962eb3f195473c4cf0c14a24fa62de524cc739866dd4482b014e0afe40bde"}
Jul 17 21:51:11.535446 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:11.535056    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-z8r8m"
Jul 17 21:51:11.536684 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:11.536657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr" event={"ID":"81716365-9c9f-4819-b6d4-e948453cb094","Type":"ContainerStarted","Data":"51c8972b2273c0ddbb03cbc628dbd7af6eff12a29e00ab68a9e16b3734862278"}
Jul 17 21:51:11.551584 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:11.551534    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-z8r8m" podStartSLOduration=2.700527354 podStartE2EDuration="5.55151816s" totalImagesPullingTime="2.850990806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:11.550644859 +0000 UTC m=+334.939661511" watchObservedRunningTime="2026-07-17 21:51:11.55151816 +0000 UTC m=+334.940534835"
Jul 17 21:51:11.568052 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:11.567998    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85fdd95b7f-x9zvr" podStartSLOduration=1.729852186 podStartE2EDuration="5.567981659s" totalImagesPullingTime="3.838129473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:11.566733502 +0000 UTC m=+334.955758223" watchObservedRunningTime="2026-07-17 21:51:11.567981659 +0000 UTC m=+334.956998309"
Jul 17 21:51:12.422773 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.422739    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:12.521001 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.520966    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:12.673371 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.673278    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"]
Jul 17 21:51:12.676677 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.676653    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.678981 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.678956    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 21:51:12.679122 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.679019    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-kgf5f\""
Jul 17 21:51:12.687744 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.687718    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"]
Jul 17 21:51:12.712019 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.711991    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:12.725689 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.725659    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"]
Jul 17 21:51:12.728967 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.728949    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.741262 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.741235    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"]
Jul 17 21:51:12.743103 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.743057    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckqvm\" (UniqueName: \"kubernetes.io/projected/d879944f-aa6b-44e9-9418-a0fc9873c92c-kube-api-access-ckqvm\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.743192 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.743130    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.743192 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.743167    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.743294 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.743276    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d879944f-aa6b-44e9-9418-a0fc9873c92c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.743367 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.743338    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.743485 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.743384    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.743485 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.743410    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.743485 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.743448    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.743485 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.743474    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.844263 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844218    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/314b915f-30de-4d8d-af9e-01f8d2b7fd08-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.844460 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d879944f-aa6b-44e9-9418-a0fc9873c92c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.844460 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844340    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgzfk\" (UniqueName: \"kubernetes.io/projected/314b915f-30de-4d8d-af9e-01f8d2b7fd08-kube-api-access-rgzfk\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.844460 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/314b915f-30de-4d8d-af9e-01f8d2b7fd08-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.844460 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844451    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/314b915f-30de-4d8d-af9e-01f8d2b7fd08-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.844679 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844470    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/314b915f-30de-4d8d-af9e-01f8d2b7fd08-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.844679 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.844679 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844522    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.844679 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844549    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.844679 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.844679 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844639    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.844679 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844667    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/314b915f-30de-4d8d-af9e-01f8d2b7fd08-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.845026 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844710    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqvm\" (UniqueName: \"kubernetes.io/projected/d879944f-aa6b-44e9-9418-a0fc9873c92c-kube-api-access-ckqvm\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.845026 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/314b915f-30de-4d8d-af9e-01f8d2b7fd08-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.845026 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844767    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/314b915f-30de-4d8d-af9e-01f8d2b7fd08-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.845026 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844799    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.845026 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/314b915f-30de-4d8d-af9e-01f8d2b7fd08-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.845026 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844843    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.845026 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844882    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.845026 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.844897    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.845357 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.845042    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d879944f-aa6b-44e9-9418-a0fc9873c92c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.845357 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.845343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.845449 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.845362    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.847310 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.847283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.847500 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.847481    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.852645 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.852620    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.852785 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.852763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckqvm\" (UniqueName: \"kubernetes.io/projected/d879944f-aa6b-44e9-9418-a0fc9873c92c-kube-api-access-ckqvm\") pod \"data-science-gateway-data-science-gateway-class-548557588-crmnj\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.945822 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.945733    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/314b915f-30de-4d8d-af9e-01f8d2b7fd08-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.945822 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.945777    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/314b915f-30de-4d8d-af9e-01f8d2b7fd08-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.945822 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.945801    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/314b915f-30de-4d8d-af9e-01f8d2b7fd08-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.946171 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.945858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/314b915f-30de-4d8d-af9e-01f8d2b7fd08-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.946171 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.945896    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgzfk\" (UniqueName: \"kubernetes.io/projected/314b915f-30de-4d8d-af9e-01f8d2b7fd08-kube-api-access-rgzfk\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.946171 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.945944    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/314b915f-30de-4d8d-af9e-01f8d2b7fd08-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.946171 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.945966    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/314b915f-30de-4d8d-af9e-01f8d2b7fd08-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.946171 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.945994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/314b915f-30de-4d8d-af9e-01f8d2b7fd08-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.946171 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.946053    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/314b915f-30de-4d8d-af9e-01f8d2b7fd08-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.946480 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.946217    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/314b915f-30de-4d8d-af9e-01f8d2b7fd08-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.946480 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.946254    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/314b915f-30de-4d8d-af9e-01f8d2b7fd08-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.946480 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.946445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/314b915f-30de-4d8d-af9e-01f8d2b7fd08-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.946793 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.946530    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/314b915f-30de-4d8d-af9e-01f8d2b7fd08-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.946793 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.946767    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/314b915f-30de-4d8d-af9e-01f8d2b7fd08-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.948306 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.948260    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/314b915f-30de-4d8d-af9e-01f8d2b7fd08-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.948575 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.948547    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/314b915f-30de-4d8d-af9e-01f8d2b7fd08-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.955709 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.955675    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/314b915f-30de-4d8d-af9e-01f8d2b7fd08-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.955834 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.955793    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgzfk\" (UniqueName: \"kubernetes.io/projected/314b915f-30de-4d8d-af9e-01f8d2b7fd08-kube-api-access-rgzfk\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6782hq7p\" (UID: \"314b915f-30de-4d8d-af9e-01f8d2b7fd08\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:12.987903 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.987856    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:12.994525 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:12.994482    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:13.039798 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:13.039720    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:13.096318 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:13.096242    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:13.136736 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:13.136702    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"]
Jul 17 21:51:13.140234 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:51:13.140191    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd879944f_aa6b_44e9_9418_a0fc9873c92c.slice/crio-73eccdf00282792d5c1e617ee25f489d9757a83751ecdec9d3f2fde88696a2e1 WatchSource:0}: Error finding container 73eccdf00282792d5c1e617ee25f489d9757a83751ecdec9d3f2fde88696a2e1: Status 404 returned error can't find the container with id 73eccdf00282792d5c1e617ee25f489d9757a83751ecdec9d3f2fde88696a2e1
Jul 17 21:51:13.186619 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:13.186593    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"]
Jul 17 21:51:13.190235 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:51:13.190209    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314b915f_30de_4d8d_af9e_01f8d2b7fd08.slice/crio-e701593ddd304d23c5c22c9a2d44c0466c23ee05689548affb4eec3973d44d33 WatchSource:0}: Error finding container e701593ddd304d23c5c22c9a2d44c0466c23ee05689548affb4eec3973d44d33: Status 404 returned error can't find the container with id e701593ddd304d23c5c22c9a2d44c0466c23ee05689548affb4eec3973d44d33
Jul 17 21:51:13.546002 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:13.545954    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p" event={"ID":"314b915f-30de-4d8d-af9e-01f8d2b7fd08","Type":"ContainerStarted","Data":"e701593ddd304d23c5c22c9a2d44c0466c23ee05689548affb4eec3973d44d33"}
Jul 17 21:51:13.547119 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:13.547100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj" event={"ID":"d879944f-aa6b-44e9-9418-a0fc9873c92c","Type":"ContainerStarted","Data":"73eccdf00282792d5c1e617ee25f489d9757a83751ecdec9d3f2fde88696a2e1"}
Jul 17 21:51:15.981520 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:15.981477    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 21:51:15.981858 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:15.981561    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 21:51:15.981858 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:15.981597    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 21:51:15.988906 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:15.988875    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 21:51:15.988979 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:15.988943    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 21:51:15.988979 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:15.988970    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 21:51:16.563692 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.563654    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p" event={"ID":"314b915f-30de-4d8d-af9e-01f8d2b7fd08","Type":"ContainerStarted","Data":"bb070873bc2455bebd4821275ffd8654a6b39d05b39e27d5e441b1c4011e6ac8"}
Jul 17 21:51:16.565211 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.565187    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj" event={"ID":"d879944f-aa6b-44e9-9418-a0fc9873c92c","Type":"ContainerStarted","Data":"c3221fd391e74b7a2a923d0831e89f87e2f4ac2c7fc74e553fbde163c6baf0b3"}
Jul 17 21:51:16.608446 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.608391    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p" podStartSLOduration=1.81979263 podStartE2EDuration="4.608372444s" totalImagesPullingTime="2.788579814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:16.606756601 +0000 UTC m=+339.995773252" watchObservedRunningTime="2026-07-17 21:51:16.608372444 +0000 UTC m=+339.997389096"
Jul 17 21:51:16.644711 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.644657    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj" podStartSLOduration=1.798592314 podStartE2EDuration="4.644639396s" totalImagesPullingTime="2.846047082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:16.643494105 +0000 UTC m=+340.032510757" watchObservedRunningTime="2026-07-17 21:51:16.644639396 +0000 UTC m=+340.033656047"
Jul 17 21:51:16.707780 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.707748    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"]
Jul 17 21:51:16.710631 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.710613    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.713181 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.713157    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 17 21:51:16.713310 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.713188    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 17 21:51:16.713310 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.713254    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 17 21:51:16.713614 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.713599    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4qrs7\""
Jul 17 21:51:16.727234 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.727202    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"]
Jul 17 21:51:16.784712 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.784667    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.784712 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.784715    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp4p6\" (UniqueName: \"kubernetes.io/projected/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-kube-api-access-vp4p6\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.784935 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.784753    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.784935 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.784779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.784935 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.784814    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.784935 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.784844    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.785056 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.784933    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.886292 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.886198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.886292 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.886278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.886503 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.886309    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp4p6\" (UniqueName: \"kubernetes.io/projected/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-kube-api-access-vp4p6\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.886503 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.886337    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.886503 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.886377    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.886503 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.886434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.886503 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.886457    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.887135 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.887104    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.888965 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.888943    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.889061 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.889025    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.889374 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.889353    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.889481 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.889465    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.895524 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.895499    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.895928 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.895905    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp4p6\" (UniqueName: \"kubernetes.io/projected/b33878cc-2675-4a04-ad0d-7a2cf0d4db99-kube-api-access-vp4p6\") pod \"istiod-openshift-gateway-94bb8fbfd-8l8sd\" (UID: \"b33878cc-2675-4a04-ad0d-7a2cf0d4db99\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:16.988035 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:16.987999    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:17.019597 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:17.019560    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:17.040188 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:17.040156    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:17.045483 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:17.045459    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:17.207947 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:17.207854    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"]
Jul 17 21:51:17.213923 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:51:17.213892    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33878cc_2675_4a04_ad0d_7a2cf0d4db99.slice/crio-fbaa296b3d79148c03b7f5b9a46b6512d7c2982ed4a1d56b5e14dd093bfa485c WatchSource:0}: Error finding container fbaa296b3d79148c03b7f5b9a46b6512d7c2982ed4a1d56b5e14dd093bfa485c: Status 404 returned error can't find the container with id fbaa296b3d79148c03b7f5b9a46b6512d7c2982ed4a1d56b5e14dd093bfa485c
Jul 17 21:51:17.577035 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:17.576993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd" event={"ID":"b33878cc-2675-4a04-ad0d-7a2cf0d4db99","Type":"ContainerStarted","Data":"fbaa296b3d79148c03b7f5b9a46b6512d7c2982ed4a1d56b5e14dd093bfa485c"}
Jul 17 21:51:17.577325 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:17.577302    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:17.578312 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:17.578292    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6782hq7p"
Jul 17 21:51:17.722204 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:17.722170    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"]
Jul 17 21:51:17.990398 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:17.990307    2565 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-548557588-crmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Jul 17 21:51:17.990398 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:17.990361    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj" podUID="d879944f-aa6b-44e9-9418-a0fc9873c92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Jul 17 21:51:18.207313 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.207234    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:18.352054 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.352022    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r56kt"]
Jul 17 21:51:18.355617 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.355594    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r56kt"
Jul 17 21:51:18.358508 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.358480    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-th52d\""
Jul 17 21:51:18.361045 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.361024    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 17 21:51:18.382128 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.380087    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r56kt"]
Jul 17 21:51:18.388693 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.388571    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:18.432500 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.432466    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5fl2x"]
Jul 17 21:51:18.436129 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.436104    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5fl2x"
Jul 17 21:51:18.439003 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.438979    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-ghwdr\""
Jul 17 21:51:18.439479 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.439460    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 17 21:51:18.456379 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.456306    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5fl2x"]
Jul 17 21:51:18.500905 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.500865    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7zpr\" (UniqueName: \"kubernetes.io/projected/1416461e-cf9d-4168-aca9-b496c64bcf7c-kube-api-access-r7zpr\") pod \"kserve-controller-manager-d6f4b65b8-r56kt\" (UID: \"1416461e-cf9d-4168-aca9-b496c64bcf7c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r56kt"
Jul 17 21:51:18.501101 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.501019    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1416461e-cf9d-4168-aca9-b496c64bcf7c-cert\") pod \"kserve-controller-manager-d6f4b65b8-r56kt\" (UID: \"1416461e-cf9d-4168-aca9-b496c64bcf7c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r56kt"
Jul 17 21:51:18.601669 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.601636    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7zpr\" (UniqueName: \"kubernetes.io/projected/1416461e-cf9d-4168-aca9-b496c64bcf7c-kube-api-access-r7zpr\") pod \"kserve-controller-manager-d6f4b65b8-r56kt\" (UID: \"1416461e-cf9d-4168-aca9-b496c64bcf7c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r56kt"
Jul 17 21:51:18.601848 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.601741    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65xmc\" (UniqueName: \"kubernetes.io/projected/d2089a42-8dbb-413a-b6cf-749899b408c0-kube-api-access-65xmc\") pod \"llmisvc-controller-manager-777c6bd7d5-5fl2x\" (UID: \"d2089a42-8dbb-413a-b6cf-749899b408c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5fl2x"
Jul 17 21:51:18.601848 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.601825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1416461e-cf9d-4168-aca9-b496c64bcf7c-cert\") pod \"kserve-controller-manager-d6f4b65b8-r56kt\" (UID: \"1416461e-cf9d-4168-aca9-b496c64bcf7c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r56kt"
Jul 17 21:51:18.601980 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.601846    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2089a42-8dbb-413a-b6cf-749899b408c0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5fl2x\" (UID: \"d2089a42-8dbb-413a-b6cf-749899b408c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5fl2x"
Jul 17 21:51:18.604579 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.604508    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:18.604696 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.604678    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1416461e-cf9d-4168-aca9-b496c64bcf7c-cert\") pod \"kserve-controller-manager-d6f4b65b8-r56kt\" (UID: \"1416461e-cf9d-4168-aca9-b496c64bcf7c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r56kt"
Jul 17 21:51:18.623923 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.623894    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7zpr\" (UniqueName: \"kubernetes.io/projected/1416461e-cf9d-4168-aca9-b496c64bcf7c-kube-api-access-r7zpr\") pod \"kserve-controller-manager-d6f4b65b8-r56kt\" (UID: \"1416461e-cf9d-4168-aca9-b496c64bcf7c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r56kt"
Jul 17 21:51:18.702748 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.702713    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2089a42-8dbb-413a-b6cf-749899b408c0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5fl2x\" (UID: \"d2089a42-8dbb-413a-b6cf-749899b408c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5fl2x"
Jul 17 21:51:18.703345 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.703312    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65xmc\" (UniqueName: \"kubernetes.io/projected/d2089a42-8dbb-413a-b6cf-749899b408c0-kube-api-access-65xmc\") pod \"llmisvc-controller-manager-777c6bd7d5-5fl2x\" (UID: \"d2089a42-8dbb-413a-b6cf-749899b408c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5fl2x"
Jul 17 21:51:18.706144 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.706121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2089a42-8dbb-413a-b6cf-749899b408c0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5fl2x\" (UID: \"d2089a42-8dbb-413a-b6cf-749899b408c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5fl2x"
Jul 17 21:51:18.706267 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.706239    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r56kt"
Jul 17 21:51:18.728123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.728088    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65xmc\" (UniqueName: \"kubernetes.io/projected/d2089a42-8dbb-413a-b6cf-749899b408c0-kube-api-access-65xmc\") pod \"llmisvc-controller-manager-777c6bd7d5-5fl2x\" (UID: \"d2089a42-8dbb-413a-b6cf-749899b408c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5fl2x"
Jul 17 21:51:18.747849 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.747816    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5fl2x"
Jul 17 21:51:18.787483 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.787411    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:18.892253 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.892143    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r56kt"]
Jul 17 21:51:18.935651 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.935625    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5fl2x"]
Jul 17 21:51:18.988754 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.988725    2565 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-548557588-crmnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Jul 17 21:51:18.988926 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:18.988777    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj" podUID="d879944f-aa6b-44e9-9418-a0fc9873c92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Jul 17 21:51:19.586761 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:19.586718    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj" podUID="d879944f-aa6b-44e9-9418-a0fc9873c92c" containerName="istio-proxy" containerID="cri-o://c3221fd391e74b7a2a923d0831e89f87e2f4ac2c7fc74e553fbde163c6baf0b3" gracePeriod=30
Jul 17 21:51:19.625582 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:19.625531    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:19.699016 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:51:19.698973    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1416461e_cf9d_4168_aca9_b496c64bcf7c.slice/crio-c5440800908b81a6a467ffee20a7acfd0c89cf70f14d4d7cdc4ec6e1b2241a94 WatchSource:0}: Error finding container c5440800908b81a6a467ffee20a7acfd0c89cf70f14d4d7cdc4ec6e1b2241a94: Status 404 returned error can't find the container with id c5440800908b81a6a467ffee20a7acfd0c89cf70f14d4d7cdc4ec6e1b2241a94
Jul 17 21:51:19.699622 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:51:19.699606    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd2089a42_8dbb_413a_b6cf_749899b408c0.slice/crio-d4e86d719ee43f87aa18ba3f7632e6d9c68ab41351ad11bfabe2bb0d4ad97f11 WatchSource:0}: Error finding container d4e86d719ee43f87aa18ba3f7632e6d9c68ab41351ad11bfabe2bb0d4ad97f11: Status 404 returned error can't find the container with id d4e86d719ee43f87aa18ba3f7632e6d9c68ab41351ad11bfabe2bb0d4ad97f11
Jul 17 21:51:19.724358 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:19.724334    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:19.757679 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:19.757648    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 21:51:19.757745 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:19.757718    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 21:51:19.829895 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:19.829859    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:20.594533 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:20.594501    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r56kt" event={"ID":"1416461e-cf9d-4168-aca9-b496c64bcf7c","Type":"ContainerStarted","Data":"c5440800908b81a6a467ffee20a7acfd0c89cf70f14d4d7cdc4ec6e1b2241a94"}
Jul 17 21:51:20.596788 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:20.596754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5fl2x" event={"ID":"d2089a42-8dbb-413a-b6cf-749899b408c0","Type":"ContainerStarted","Data":"d4e86d719ee43f87aa18ba3f7632e6d9c68ab41351ad11bfabe2bb0d4ad97f11"}
Jul 17 21:51:20.599607 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:20.599575    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd" event={"ID":"b33878cc-2675-4a04-ad0d-7a2cf0d4db99","Type":"ContainerStarted","Data":"172dad431a67248a6206180be5966e0e9ca31d294382ec1cf0ef673b988dec78"}
Jul 17 21:51:20.600008 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:20.599955    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:20.601938 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:20.601912    2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-8l8sd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 21:51:20.602099 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:20.602002    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd" podUID="b33878cc-2675-4a04-ad0d-7a2cf0d4db99" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 21:51:20.655134 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:20.654843    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd" podStartSLOduration=2.113596913 podStartE2EDuration="4.654813063s" totalImagesPullingTime="2.54121615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:20.650009077 +0000 UTC m=+344.039025754" watchObservedRunningTime="2026-07-17 21:51:20.654813063 +0000 UTC m=+344.043829716"
Jul 17 21:51:21.605050 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:21.605018    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-8l8sd"
Jul 17 21:51:22.543439 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:22.543409    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-z8r8m"
Jul 17 21:51:24.619759 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.619725    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r56kt" event={"ID":"1416461e-cf9d-4168-aca9-b496c64bcf7c","Type":"ContainerStarted","Data":"0381eb9d9b5e5dc570d42e56423de4c5b289e566ab7d8589722de2f7c2e09cff"}
Jul 17 21:51:24.620256 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.619835    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r56kt"
Jul 17 21:51:24.621254 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.621233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5fl2x" event={"ID":"d2089a42-8dbb-413a-b6cf-749899b408c0","Type":"ContainerStarted","Data":"00235bd558034803da7d32be396f72783ee5660a5f004e99cd8f205707fc9247"}
Jul 17 21:51:24.621385 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.621369    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5fl2x"
Jul 17 21:51:24.640032 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.639974    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r56kt" podStartSLOduration=2.83177231 podStartE2EDuration="6.639957574s" totalImagesPullingTime="3.808185264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:24.638382667 +0000 UTC m=+348.027399339" watchObservedRunningTime="2026-07-17 21:51:24.639957574 +0000 UTC m=+348.028974225"
Jul 17 21:51:24.658670 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.658601    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5fl2x" podStartSLOduration=2.845156255 podStartE2EDuration="6.658578434s" totalImagesPullingTime="3.813422179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:24.656347377 +0000 UTC m=+348.045364054" watchObservedRunningTime="2026-07-17 21:51:24.658578434 +0000 UTC m=+348.047595085"
Jul 17 21:51:24.841611 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.841588    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:24.967003 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.966909    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d879944f-aa6b-44e9-9418-a0fc9873c92c-istiod-ca-cert\") pod \"d879944f-aa6b-44e9-9418-a0fc9873c92c\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") "
Jul 17 21:51:24.967003 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.966957    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-workload-socket\") pod \"d879944f-aa6b-44e9-9418-a0fc9873c92c\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") "
Jul 17 21:51:24.967003 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.966984    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-workload-certs\") pod \"d879944f-aa6b-44e9-9418-a0fc9873c92c\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") "
Jul 17 21:51:24.967317 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.967013    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-credential-socket\") pod \"d879944f-aa6b-44e9-9418-a0fc9873c92c\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") "
Jul 17 21:51:24.967317 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.967042    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-token\") pod \"d879944f-aa6b-44e9-9418-a0fc9873c92c\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") "
Jul 17 21:51:24.967317 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.967119    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckqvm\" (UniqueName: \"kubernetes.io/projected/d879944f-aa6b-44e9-9418-a0fc9873c92c-kube-api-access-ckqvm\") pod \"d879944f-aa6b-44e9-9418-a0fc9873c92c\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") "
Jul 17 21:51:24.967317 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.967143    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-envoy\") pod \"d879944f-aa6b-44e9-9418-a0fc9873c92c\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") "
Jul 17 21:51:24.967317 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.967167    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-data\") pod \"d879944f-aa6b-44e9-9418-a0fc9873c92c\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") "
Jul 17 21:51:24.967317 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.967215    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-podinfo\") pod \"d879944f-aa6b-44e9-9418-a0fc9873c92c\" (UID: \"d879944f-aa6b-44e9-9418-a0fc9873c92c\") "
Jul 17 21:51:24.967596 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.967367    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d879944f-aa6b-44e9-9418-a0fc9873c92c" (UID: "d879944f-aa6b-44e9-9418-a0fc9873c92c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:51:24.967596 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.967410    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d879944f-aa6b-44e9-9418-a0fc9873c92c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d879944f-aa6b-44e9-9418-a0fc9873c92c" (UID: "d879944f-aa6b-44e9-9418-a0fc9873c92c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:51:24.967596 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.967409    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d879944f-aa6b-44e9-9418-a0fc9873c92c" (UID: "d879944f-aa6b-44e9-9418-a0fc9873c92c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:51:24.967749 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.967621    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-data" (OuterVolumeSpecName: "istio-data") pod "d879944f-aa6b-44e9-9418-a0fc9873c92c" (UID: "d879944f-aa6b-44e9-9418-a0fc9873c92c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:51:24.967749 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.967650    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d879944f-aa6b-44e9-9418-a0fc9873c92c-istiod-ca-cert\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:51:24.967749 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.967667    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-workload-certs\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:51:24.967749 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.967681    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-credential-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:51:24.967749 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.967697    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d879944f-aa6b-44e9-9418-a0fc9873c92c" (UID: "d879944f-aa6b-44e9-9418-a0fc9873c92c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:51:24.970051 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.969999    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d879944f-aa6b-44e9-9418-a0fc9873c92c-kube-api-access-ckqvm" (OuterVolumeSpecName: "kube-api-access-ckqvm") pod "d879944f-aa6b-44e9-9418-a0fc9873c92c" (UID: "d879944f-aa6b-44e9-9418-a0fc9873c92c"). InnerVolumeSpecName "kube-api-access-ckqvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:51:24.970223 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.970158    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d879944f-aa6b-44e9-9418-a0fc9873c92c" (UID: "d879944f-aa6b-44e9-9418-a0fc9873c92c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:51:24.970223 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.970174    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-token" (OuterVolumeSpecName: "istio-token") pod "d879944f-aa6b-44e9-9418-a0fc9873c92c" (UID: "d879944f-aa6b-44e9-9418-a0fc9873c92c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:51:24.970223 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:24.970200    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d879944f-aa6b-44e9-9418-a0fc9873c92c" (UID: "d879944f-aa6b-44e9-9418-a0fc9873c92c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 21:51:25.069052 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:25.069015    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckqvm\" (UniqueName: \"kubernetes.io/projected/d879944f-aa6b-44e9-9418-a0fc9873c92c-kube-api-access-ckqvm\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:51:25.069052 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:25.069049    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-envoy\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:51:25.069052 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:25.069058    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-data\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:51:25.069363 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:25.069087    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-podinfo\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:51:25.069363 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:25.069096    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d879944f-aa6b-44e9-9418-a0fc9873c92c-workload-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:51:25.069363 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:25.069103    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d879944f-aa6b-44e9-9418-a0fc9873c92c-istio-token\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:51:25.627288 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:25.627246    2565 generic.go:356] "Generic (PLEG): container finished" podID="d879944f-aa6b-44e9-9418-a0fc9873c92c" containerID="c3221fd391e74b7a2a923d0831e89f87e2f4ac2c7fc74e553fbde163c6baf0b3" exitCode=0
Jul 17 21:51:25.627734 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:25.627298    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"
Jul 17 21:51:25.627734 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:25.627330    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj" event={"ID":"d879944f-aa6b-44e9-9418-a0fc9873c92c","Type":"ContainerDied","Data":"c3221fd391e74b7a2a923d0831e89f87e2f4ac2c7fc74e553fbde163c6baf0b3"}
Jul 17 21:51:25.627734 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:25.627385    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj" event={"ID":"d879944f-aa6b-44e9-9418-a0fc9873c92c","Type":"ContainerDied","Data":"73eccdf00282792d5c1e617ee25f489d9757a83751ecdec9d3f2fde88696a2e1"}
Jul 17 21:51:25.627734 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:25.627407    2565 scope.go:117] "RemoveContainer" containerID="c3221fd391e74b7a2a923d0831e89f87e2f4ac2c7fc74e553fbde163c6baf0b3"
Jul 17 21:51:25.636370 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:25.636352    2565 scope.go:117] "RemoveContainer" containerID="c3221fd391e74b7a2a923d0831e89f87e2f4ac2c7fc74e553fbde163c6baf0b3"
Jul 17 21:51:25.636680 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:51:25.636661    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3221fd391e74b7a2a923d0831e89f87e2f4ac2c7fc74e553fbde163c6baf0b3\": container with ID starting with c3221fd391e74b7a2a923d0831e89f87e2f4ac2c7fc74e553fbde163c6baf0b3 not found: ID does not exist" containerID="c3221fd391e74b7a2a923d0831e89f87e2f4ac2c7fc74e553fbde163c6baf0b3"
Jul 17 21:51:25.636737 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:25.636690    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3221fd391e74b7a2a923d0831e89f87e2f4ac2c7fc74e553fbde163c6baf0b3"} err="failed to get container status \"c3221fd391e74b7a2a923d0831e89f87e2f4ac2c7fc74e553fbde163c6baf0b3\": rpc error: code = NotFound desc = could not find container \"c3221fd391e74b7a2a923d0831e89f87e2f4ac2c7fc74e553fbde163c6baf0b3\": container with ID starting with c3221fd391e74b7a2a923d0831e89f87e2f4ac2c7fc74e553fbde163c6baf0b3 not found: ID does not exist"
Jul 17 21:51:25.648144 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:25.648120    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"]
Jul 17 21:51:25.656207 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:25.656183    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-548557588-crmnj"]
Jul 17 21:51:27.182440 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:27.182410    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d879944f-aa6b-44e9-9418-a0fc9873c92c" path="/var/lib/kubelet/pods/d879944f-aa6b-44e9-9418-a0fc9873c92c/volumes"
Jul 17 21:51:33.829197 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:33.829112    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:33.929158 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:33.929117    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:33.982869 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:33.982827    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:51:55.629849 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:55.629814    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5fl2x"
Jul 17 21:51:55.632724 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:55.632706    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r56kt"
Jul 17 21:51:56.040737 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:51:56.040696    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:07.621036 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:07.620998    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:07.720228 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:07.720182    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:07.818247 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:07.818215    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:08.663701 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:08.663659    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:08.750638 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:08.750602    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:37.063317 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:37.063284    2565 scope.go:117] "RemoveContainer" containerID="228a8a67d2c9829792889f63ab3498396e079ee53a3643f43dc31f41a0828765"
Jul 17 21:52:44.395043 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:44.395007    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:48.185649 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:48.185607    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:49.014811 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:49.014775    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:52.383431 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:52.383389    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:53.433200 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:53.433164    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:53.534252 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:53.534220    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:55.221408 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:55.221373    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:56.649739 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:56.649697    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:57.789424 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:57.789389    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:58.615920 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:58.615885    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:52:59.892131 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:59.892093    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"]
Jul 17 21:52:59.892523 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:59.892466    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d879944f-aa6b-44e9-9418-a0fc9873c92c" containerName="istio-proxy"
Jul 17 21:52:59.892523 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:59.892481    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d879944f-aa6b-44e9-9418-a0fc9873c92c" containerName="istio-proxy"
Jul 17 21:52:59.892595 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:59.892546    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d879944f-aa6b-44e9-9418-a0fc9873c92c" containerName="istio-proxy"
Jul 17 21:52:59.895290 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:59.895272    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"
Jul 17 21:52:59.897988 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:59.897961    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dwjnc\""
Jul 17 21:52:59.898179 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:59.898043    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 21:52:59.898667 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:59.898652    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 21:52:59.910482 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:52:59.910454    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"]
Jul 17 21:53:00.000786 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:00.000750    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9z94\" (UniqueName: \"kubernetes.io/projected/d31db8e6-1237-4ad5-853a-ef7d2079e73d-kube-api-access-h9z94\") pod \"limitador-operator-controller-manager-859d7b78bd-glchz\" (UID: \"d31db8e6-1237-4ad5-853a-ef7d2079e73d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"
Jul 17 21:53:00.101835 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:00.101802    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9z94\" (UniqueName: \"kubernetes.io/projected/d31db8e6-1237-4ad5-853a-ef7d2079e73d-kube-api-access-h9z94\") pod \"limitador-operator-controller-manager-859d7b78bd-glchz\" (UID: \"d31db8e6-1237-4ad5-853a-ef7d2079e73d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"
Jul 17 21:53:00.110728 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:00.110700    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9z94\" (UniqueName: \"kubernetes.io/projected/d31db8e6-1237-4ad5-853a-ef7d2079e73d-kube-api-access-h9z94\") pod \"limitador-operator-controller-manager-859d7b78bd-glchz\" (UID: \"d31db8e6-1237-4ad5-853a-ef7d2079e73d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"
Jul 17 21:53:00.205360 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:00.205264    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"
Jul 17 21:53:00.337886 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:00.337805    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"]
Jul 17 21:53:00.342920 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:53:00.342895    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31db8e6_1237_4ad5_853a_ef7d2079e73d.slice/crio-beedaa3873047cbf9ff780964651fb677155ac18a5ec3153b009b55dafa3b0a1 WatchSource:0}: Error finding container beedaa3873047cbf9ff780964651fb677155ac18a5ec3153b009b55dafa3b0a1: Status 404 returned error can't find the container with id beedaa3873047cbf9ff780964651fb677155ac18a5ec3153b009b55dafa3b0a1
Jul 17 21:53:01.035710 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:01.035662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz" event={"ID":"d31db8e6-1237-4ad5-853a-ef7d2079e73d","Type":"ContainerStarted","Data":"beedaa3873047cbf9ff780964651fb677155ac18a5ec3153b009b55dafa3b0a1"}
Jul 17 21:53:01.206485 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:01.206425    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:01.307622 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:01.307428    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:03.046277 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:03.046241    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz" event={"ID":"d31db8e6-1237-4ad5-853a-ef7d2079e73d","Type":"ContainerStarted","Data":"836eee05570d8c323af3ebdf0ba277e7bcdd50a21ab7345e2f43d0619b22d71d"}
Jul 17 21:53:03.046780 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:03.046383    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"
Jul 17 21:53:03.062648 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:03.062595    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz" podStartSLOduration=2.336820155 podStartE2EDuration="4.062576874s" totalImagesPullingTime="1.725756719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:03.061332804 +0000 UTC m=+446.450349455" watchObservedRunningTime="2026-07-17 21:53:03.062576874 +0000 UTC m=+446.451593525"
Jul 17 21:53:04.941123 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:04.941086    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:05.201315 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:05.201220    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:06.800288 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:06.800247    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:06.917497 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:06.917465    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:07.184376 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:07.184289    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:07.246327 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:07.246292    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:10.406626 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:10.406596    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:10.792409 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:10.792378    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:10.893151 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:10.893114    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:14.052988 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:14.052961    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"
Jul 17 21:53:15.519963 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.519923    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"]
Jul 17 21:53:15.520494 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.520201    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz" podUID="d31db8e6-1237-4ad5-853a-ef7d2079e73d" containerName="manager" containerID="cri-o://836eee05570d8c323af3ebdf0ba277e7bcdd50a21ab7345e2f43d0619b22d71d" gracePeriod=2
Jul 17 21:53:15.522243 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:53:15.522211    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-glchz\" is forbidden: User \"system:node:ip-10-0-129-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-11.ec2.internal' and this object" podUID="d31db8e6-1237-4ad5-853a-ef7d2079e73d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"
Jul 17 21:53:15.522677 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.522654    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"]
Jul 17 21:53:15.545093 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.545033    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sqgcc"]
Jul 17 21:53:15.545406 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.545392    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d31db8e6-1237-4ad5-853a-ef7d2079e73d" containerName="manager"
Jul 17 21:53:15.545451 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.545408    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d31db8e6-1237-4ad5-853a-ef7d2079e73d" containerName="manager"
Jul 17 21:53:15.545483 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.545459    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d31db8e6-1237-4ad5-853a-ef7d2079e73d" containerName="manager"
Jul 17 21:53:15.548373 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.548349    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sqgcc"
Jul 17 21:53:15.550683 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:53:15.550645    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-glchz\" is forbidden: User \"system:node:ip-10-0-129-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-11.ec2.internal' and this object" podUID="d31db8e6-1237-4ad5-853a-ef7d2079e73d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"
Jul 17 21:53:15.559516 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.559485    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sqgcc"]
Jul 17 21:53:15.618088 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.618035    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwj6n\" (UniqueName: \"kubernetes.io/projected/5d5837b7-e157-4a0e-836a-fab4f0bd6ddf-kube-api-access-gwj6n\") pod \"limitador-operator-controller-manager-859d7b78bd-sqgcc\" (UID: \"5d5837b7-e157-4a0e-836a-fab4f0bd6ddf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sqgcc"
Jul 17 21:53:15.719425 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.719383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwj6n\" (UniqueName: \"kubernetes.io/projected/5d5837b7-e157-4a0e-836a-fab4f0bd6ddf-kube-api-access-gwj6n\") pod \"limitador-operator-controller-manager-859d7b78bd-sqgcc\" (UID: \"5d5837b7-e157-4a0e-836a-fab4f0bd6ddf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sqgcc"
Jul 17 21:53:15.728666 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.728629    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwj6n\" (UniqueName: \"kubernetes.io/projected/5d5837b7-e157-4a0e-836a-fab4f0bd6ddf-kube-api-access-gwj6n\") pod \"limitador-operator-controller-manager-859d7b78bd-sqgcc\" (UID: \"5d5837b7-e157-4a0e-836a-fab4f0bd6ddf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sqgcc"
Jul 17 21:53:15.766880 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.766851    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"
Jul 17 21:53:15.768839 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:53:15.768803    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-glchz\" is forbidden: User \"system:node:ip-10-0-129-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-11.ec2.internal' and this object" podUID="d31db8e6-1237-4ad5-853a-ef7d2079e73d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"
Jul 17 21:53:15.820116 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.819991    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9z94\" (UniqueName: \"kubernetes.io/projected/d31db8e6-1237-4ad5-853a-ef7d2079e73d-kube-api-access-h9z94\") pod \"d31db8e6-1237-4ad5-853a-ef7d2079e73d\" (UID: \"d31db8e6-1237-4ad5-853a-ef7d2079e73d\") "
Jul 17 21:53:15.822325 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.822287    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31db8e6-1237-4ad5-853a-ef7d2079e73d-kube-api-access-h9z94" (OuterVolumeSpecName: "kube-api-access-h9z94") pod "d31db8e6-1237-4ad5-853a-ef7d2079e73d" (UID: "d31db8e6-1237-4ad5-853a-ef7d2079e73d"). InnerVolumeSpecName "kube-api-access-h9z94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:53:15.903566 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.903517    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sqgcc"
Jul 17 21:53:15.920973 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:15.920941    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9z94\" (UniqueName: \"kubernetes.io/projected/d31db8e6-1237-4ad5-853a-ef7d2079e73d-kube-api-access-h9z94\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 21:53:16.045695 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:16.045665    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sqgcc"]
Jul 17 21:53:16.048079 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:53:16.048040    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5837b7_e157_4a0e_836a_fab4f0bd6ddf.slice/crio-dea55d92247401b44f4d959599d17b4d231eb79022552279b6f7775802c33476 WatchSource:0}: Error finding container dea55d92247401b44f4d959599d17b4d231eb79022552279b6f7775802c33476: Status 404 returned error can't find the container with id dea55d92247401b44f4d959599d17b4d231eb79022552279b6f7775802c33476
Jul 17 21:53:16.103178 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:16.103145    2565 generic.go:356] "Generic (PLEG): container finished" podID="d31db8e6-1237-4ad5-853a-ef7d2079e73d" containerID="836eee05570d8c323af3ebdf0ba277e7bcdd50a21ab7345e2f43d0619b22d71d" exitCode=0
Jul 17 21:53:16.103316 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:16.103199    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"
Jul 17 21:53:16.103316 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:16.103205    2565 scope.go:117] "RemoveContainer" containerID="836eee05570d8c323af3ebdf0ba277e7bcdd50a21ab7345e2f43d0619b22d71d"
Jul 17 21:53:16.105126 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:16.105100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sqgcc" event={"ID":"5d5837b7-e157-4a0e-836a-fab4f0bd6ddf","Type":"ContainerStarted","Data":"dea55d92247401b44f4d959599d17b4d231eb79022552279b6f7775802c33476"}
Jul 17 21:53:16.106153 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:53:16.106119    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-glchz\" is forbidden: User \"system:node:ip-10-0-129-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-11.ec2.internal' and this object" podUID="d31db8e6-1237-4ad5-853a-ef7d2079e73d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"
Jul 17 21:53:16.113186 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:16.113165    2565 scope.go:117] "RemoveContainer" containerID="836eee05570d8c323af3ebdf0ba277e7bcdd50a21ab7345e2f43d0619b22d71d"
Jul 17 21:53:16.113465 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:53:16.113449    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836eee05570d8c323af3ebdf0ba277e7bcdd50a21ab7345e2f43d0619b22d71d\": container with ID starting with 836eee05570d8c323af3ebdf0ba277e7bcdd50a21ab7345e2f43d0619b22d71d not found: ID does not exist" containerID="836eee05570d8c323af3ebdf0ba277e7bcdd50a21ab7345e2f43d0619b22d71d"
Jul 17 21:53:16.113526 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:16.113475    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836eee05570d8c323af3ebdf0ba277e7bcdd50a21ab7345e2f43d0619b22d71d"} err="failed to get container status \"836eee05570d8c323af3ebdf0ba277e7bcdd50a21ab7345e2f43d0619b22d71d\": rpc error: code = NotFound desc = could not find container \"836eee05570d8c323af3ebdf0ba277e7bcdd50a21ab7345e2f43d0619b22d71d\": container with ID starting with 836eee05570d8c323af3ebdf0ba277e7bcdd50a21ab7345e2f43d0619b22d71d not found: ID does not exist"
Jul 17 21:53:16.114574 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:53:16.114552    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-glchz\" is forbidden: User \"system:node:ip-10-0-129-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-11.ec2.internal' and this object" podUID="d31db8e6-1237-4ad5-853a-ef7d2079e73d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"
Jul 17 21:53:17.111843 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:17.111807    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sqgcc" event={"ID":"5d5837b7-e157-4a0e-836a-fab4f0bd6ddf","Type":"ContainerStarted","Data":"1ad7a3ad4433e8fc1a7459d0cbffa2fc325293638f56ba3afd75269c83cf05d3"}
Jul 17 21:53:17.112259 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:17.111936    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sqgcc"
Jul 17 21:53:17.114027 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:53:17.113994    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-glchz\" is forbidden: User \"system:node:ip-10-0-129-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-11.ec2.internal' and this object" podUID="d31db8e6-1237-4ad5-853a-ef7d2079e73d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"
Jul 17 21:53:17.134558 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:17.134505    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sqgcc" podStartSLOduration=2.1344866590000002 podStartE2EDuration="2.134486659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:17.133184682 +0000 UTC m=+460.522201342" watchObservedRunningTime="2026-07-17 21:53:17.134486659 +0000 UTC m=+460.523503310"
Jul 17 21:53:17.182163 ip-10-0-129-11 kubenswrapper[2565]: E0717 21:53:17.182117    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-glchz\" is forbidden: User \"system:node:ip-10-0-129-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-11.ec2.internal' and this object" podUID="d31db8e6-1237-4ad5-853a-ef7d2079e73d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-glchz"
Jul 17 21:53:17.183079 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:17.183040    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d31db8e6-1237-4ad5-853a-ef7d2079e73d" path="/var/lib/kubelet/pods/d31db8e6-1237-4ad5-853a-ef7d2079e73d/volumes"
Jul 17 21:53:17.275047 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:17.275006    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:17.379035 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:17.378863    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:19.829014 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:19.828979    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:19.929840 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:19.929809    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:25.238345 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:25.238306    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:25.290077 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:25.290017    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:25.627216 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:25.627183    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:25.727368 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:25.727334    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:28.118676 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:28.118643    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sqgcc"
Jul 17 21:53:29.232017 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:29.231976    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:29.289393 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:29.289360    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:29.420542 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:29.420501    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:29.560390 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:29.560359    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:35.652094 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:35.652047    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:35.756458 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:35.756424    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:43.134830 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.134789    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-568f5ff577-c6zrn"]
Jul 17 21:53:43.141142 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.141116    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.143503 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.143479    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 17 21:53:43.143809 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.143792    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 17 21:53:43.143859 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.143795    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 17 21:53:43.143859 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.143843    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 17 21:53:43.145128 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.145099    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 17 21:53:43.145229 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.145146    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hlhgr\""
Jul 17 21:53:43.152440 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.152410    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 17 21:53:43.153590 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.153558    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-568f5ff577-c6zrn"]
Jul 17 21:53:43.188132 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.188103    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:43.247614 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.247586    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:43.259853 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.259824    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43df38e3-f93b-4b13-8fb0-29359252f3ff-service-ca\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.259920 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.259877    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43df38e3-f93b-4b13-8fb0-29359252f3ff-oauth-serving-cert\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.259967 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.259931    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43df38e3-f93b-4b13-8fb0-29359252f3ff-console-oauth-config\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.259967 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.259956    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2p2w\" (UniqueName: \"kubernetes.io/projected/43df38e3-f93b-4b13-8fb0-29359252f3ff-kube-api-access-n2p2w\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.260051 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.260032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43df38e3-f93b-4b13-8fb0-29359252f3ff-console-serving-cert\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.260105 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.260090    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43df38e3-f93b-4b13-8fb0-29359252f3ff-trusted-ca-bundle\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.260145 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.260111    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43df38e3-f93b-4b13-8fb0-29359252f3ff-console-config\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.330932 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.330900    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:43.360792 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.360762    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43df38e3-f93b-4b13-8fb0-29359252f3ff-console-oauth-config\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.360917 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.360795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2p2w\" (UniqueName: \"kubernetes.io/projected/43df38e3-f93b-4b13-8fb0-29359252f3ff-kube-api-access-n2p2w\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.360917 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.360832    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43df38e3-f93b-4b13-8fb0-29359252f3ff-console-serving-cert\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.360917 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.360865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43df38e3-f93b-4b13-8fb0-29359252f3ff-trusted-ca-bundle\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.360917 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.360886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43df38e3-f93b-4b13-8fb0-29359252f3ff-console-config\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.360917 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.360906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43df38e3-f93b-4b13-8fb0-29359252f3ff-service-ca\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.361152 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.360947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43df38e3-f93b-4b13-8fb0-29359252f3ff-oauth-serving-cert\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.361733 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.361707    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43df38e3-f93b-4b13-8fb0-29359252f3ff-oauth-serving-cert\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.361894 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.361876    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43df38e3-f93b-4b13-8fb0-29359252f3ff-trusted-ca-bundle\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.361938 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.361897    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43df38e3-f93b-4b13-8fb0-29359252f3ff-console-config\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.362086 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.362044    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43df38e3-f93b-4b13-8fb0-29359252f3ff-service-ca\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.363524 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.363491    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43df38e3-f93b-4b13-8fb0-29359252f3ff-console-oauth-config\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.363647 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.363621    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43df38e3-f93b-4b13-8fb0-29359252f3ff-console-serving-cert\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.368697 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.368677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2p2w\" (UniqueName: \"kubernetes.io/projected/43df38e3-f93b-4b13-8fb0-29359252f3ff-kube-api-access-n2p2w\") pod \"console-568f5ff577-c6zrn\" (UID: \"43df38e3-f93b-4b13-8fb0-29359252f3ff\") " pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.431860 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.431778    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:43.452325 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.452293    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:43.592820 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.592794    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-568f5ff577-c6zrn"]
Jul 17 21:53:43.595659 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:53:43.595618    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43df38e3_f93b_4b13_8fb0_29359252f3ff.slice/crio-f8bf1acf7c5509010bdd79fbfdd679970a84094c9168c539616b7548fb7bf49a WatchSource:0}: Error finding container f8bf1acf7c5509010bdd79fbfdd679970a84094c9168c539616b7548fb7bf49a: Status 404 returned error can't find the container with id f8bf1acf7c5509010bdd79fbfdd679970a84094c9168c539616b7548fb7bf49a
Jul 17 21:53:43.927708 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.927612    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:43.981652 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:43.981616    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:44.238449 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:44.238361    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-568f5ff577-c6zrn" event={"ID":"43df38e3-f93b-4b13-8fb0-29359252f3ff","Type":"ContainerStarted","Data":"f42dd16d2440f7b10dbd67fb14143af8c36752c6d871d03e2b65b112154911bc"}
Jul 17 21:53:44.238449 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:44.238424    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-568f5ff577-c6zrn" event={"ID":"43df38e3-f93b-4b13-8fb0-29359252f3ff","Type":"ContainerStarted","Data":"f8bf1acf7c5509010bdd79fbfdd679970a84094c9168c539616b7548fb7bf49a"}
Jul 17 21:53:44.257059 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:44.257008    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-568f5ff577-c6zrn" podStartSLOduration=1.256989189 podStartE2EDuration="1.256989189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:44.256204269 +0000 UTC m=+487.645220939" watchObservedRunningTime="2026-07-17 21:53:44.256989189 +0000 UTC m=+487.646005841"
Jul 17 21:53:53.452439 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:53.452399    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:53.452858 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:53.452454    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:53.457259 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:53.457238    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:54.285305 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:54.285272    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-568f5ff577-c6zrn"
Jul 17 21:53:54.438686 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:54.438643    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:54.488954 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:54.488920    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:54.556695 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:54.556610    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:56.248947 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.248914    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"]
Jul 17 21:53:56.252860 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.252835    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.255412 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.255390    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xnk5q\""
Jul 17 21:53:56.262695 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.262671    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"]
Jul 17 21:53:56.275180 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.275147    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:56.384811 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.384773    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llr92\" (UniqueName: \"kubernetes.io/projected/fc686a0b-25d0-4d64-bf27-ff5ee005090b-kube-api-access-llr92\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.384811 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.384816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc686a0b-25d0-4d64-bf27-ff5ee005090b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.385033 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.384874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc686a0b-25d0-4d64-bf27-ff5ee005090b-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.385033 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.384900    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc686a0b-25d0-4d64-bf27-ff5ee005090b-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.385033 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.384944    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc686a0b-25d0-4d64-bf27-ff5ee005090b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.385033 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.384981    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc686a0b-25d0-4d64-bf27-ff5ee005090b-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.385033 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.385001    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc686a0b-25d0-4d64-bf27-ff5ee005090b-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.385033 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.385031    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc686a0b-25d0-4d64-bf27-ff5ee005090b-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.385272 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.385052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc686a0b-25d0-4d64-bf27-ff5ee005090b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.465285 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.465252    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:53:56.486533 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.486499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc686a0b-25d0-4d64-bf27-ff5ee005090b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.486706 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.486545    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc686a0b-25d0-4d64-bf27-ff5ee005090b-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.486706 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.486573    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc686a0b-25d0-4d64-bf27-ff5ee005090b-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.486706 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.486597    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc686a0b-25d0-4d64-bf27-ff5ee005090b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.486706 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.486626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc686a0b-25d0-4d64-bf27-ff5ee005090b-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.486706 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.486643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc686a0b-25d0-4d64-bf27-ff5ee005090b-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.486706 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.486668    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc686a0b-25d0-4d64-bf27-ff5ee005090b-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.486706 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.486687    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc686a0b-25d0-4d64-bf27-ff5ee005090b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.487051 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.486724    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llr92\" (UniqueName: \"kubernetes.io/projected/fc686a0b-25d0-4d64-bf27-ff5ee005090b-kube-api-access-llr92\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.487051 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.487009    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc686a0b-25d0-4d64-bf27-ff5ee005090b-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.487381 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.487358    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc686a0b-25d0-4d64-bf27-ff5ee005090b-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.487460 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.487406    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc686a0b-25d0-4d64-bf27-ff5ee005090b-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.487460 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.487416    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc686a0b-25d0-4d64-bf27-ff5ee005090b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.487460 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.487434    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc686a0b-25d0-4d64-bf27-ff5ee005090b-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.489403 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.489380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc686a0b-25d0-4d64-bf27-ff5ee005090b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.489629 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.489610    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc686a0b-25d0-4d64-bf27-ff5ee005090b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.494725 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.494702    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc686a0b-25d0-4d64-bf27-ff5ee005090b-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.494896 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.494877    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llr92\" (UniqueName: \"kubernetes.io/projected/fc686a0b-25d0-4d64-bf27-ff5ee005090b-kube-api-access-llr92\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pvfz2\" (UID: \"fc686a0b-25d0-4d64-bf27-ff5ee005090b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.563707 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.563613    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:56.704053 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.704027    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"]
Jul 17 21:53:56.705676 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:53:56.705641    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc686a0b_25d0_4d64_bf27_ff5ee005090b.slice/crio-e4bd26c84c523c3feff6e492413e92b4559488eb0e5e17495b0c8608aec2fb1e WatchSource:0}: Error finding container e4bd26c84c523c3feff6e492413e92b4559488eb0e5e17495b0c8608aec2fb1e: Status 404 returned error can't find the container with id e4bd26c84c523c3feff6e492413e92b4559488eb0e5e17495b0c8608aec2fb1e
Jul 17 21:53:56.707980 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.707944    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 21:53:56.708055 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.708018    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 21:53:56.708122 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:56.708052    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 21:53:57.296940 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:57.296903    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2" event={"ID":"fc686a0b-25d0-4d64-bf27-ff5ee005090b","Type":"ContainerStarted","Data":"161f99e0b9cf6c53b4a73a6e5bf5d0e7595f484182983407f2dd29ad3a1f3175"}
Jul 17 21:53:57.297406 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:57.296956    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2" event={"ID":"fc686a0b-25d0-4d64-bf27-ff5ee005090b","Type":"ContainerStarted","Data":"e4bd26c84c523c3feff6e492413e92b4559488eb0e5e17495b0c8608aec2fb1e"}
Jul 17 21:53:57.317874 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:57.317819    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2" podStartSLOduration=1.317802514 podStartE2EDuration="1.317802514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:57.316989653 +0000 UTC m=+500.706006328" watchObservedRunningTime="2026-07-17 21:53:57.317802514 +0000 UTC m=+500.706819165"
Jul 17 21:53:57.564681 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:57.564577    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:57.569737 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:57.569709    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:58.301602 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:58.301573    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:53:58.302733 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:53:58.302709    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:54:00.136480 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:00.136447    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:12.881144 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:12.881110    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:12.991323 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:12.991288    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:13.048246 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.048205    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:13.292625 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.292588    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 21:54:13.296177 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.296152    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-2g44p"
Jul 17 21:54:13.298646 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.298624    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4wlrl\""
Jul 17 21:54:13.298754 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.298654    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 17 21:54:13.306425 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.306402    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 21:54:13.332911 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.332873    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f5ee100-6729-445b-ac7d-60d590c960cc-config-file\") pod \"limitador-limitador-54d6d988d5-2g44p\" (UID: \"8f5ee100-6729-445b-ac7d-60d590c960cc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2g44p"
Jul 17 21:54:13.333116 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.332920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l5zr\" (UniqueName: \"kubernetes.io/projected/8f5ee100-6729-445b-ac7d-60d590c960cc-kube-api-access-8l5zr\") pod \"limitador-limitador-54d6d988d5-2g44p\" (UID: \"8f5ee100-6729-445b-ac7d-60d590c960cc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2g44p"
Jul 17 21:54:13.333174 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.333119    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 21:54:13.433907 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.433873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f5ee100-6729-445b-ac7d-60d590c960cc-config-file\") pod \"limitador-limitador-54d6d988d5-2g44p\" (UID: \"8f5ee100-6729-445b-ac7d-60d590c960cc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2g44p"
Jul 17 21:54:13.434137 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.433920    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l5zr\" (UniqueName: \"kubernetes.io/projected/8f5ee100-6729-445b-ac7d-60d590c960cc-kube-api-access-8l5zr\") pod \"limitador-limitador-54d6d988d5-2g44p\" (UID: \"8f5ee100-6729-445b-ac7d-60d590c960cc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2g44p"
Jul 17 21:54:13.434561 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.434543    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f5ee100-6729-445b-ac7d-60d590c960cc-config-file\") pod \"limitador-limitador-54d6d988d5-2g44p\" (UID: \"8f5ee100-6729-445b-ac7d-60d590c960cc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2g44p"
Jul 17 21:54:13.442010 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.441982    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l5zr\" (UniqueName: \"kubernetes.io/projected/8f5ee100-6729-445b-ac7d-60d590c960cc-kube-api-access-8l5zr\") pod \"limitador-limitador-54d6d988d5-2g44p\" (UID: \"8f5ee100-6729-445b-ac7d-60d590c960cc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2g44p"
Jul 17 21:54:13.607393 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.607301    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-2g44p"
Jul 17 21:54:13.730496 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.730431    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:13.746917 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.746770    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 21:54:13.749165 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:54:13.749139    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5ee100_6729_445b_ac7d_60d590c960cc.slice/crio-6d47abc0c651d60342661aad6ef653fe8d522a251ce335ece278d5f6dfa8b65f WatchSource:0}: Error finding container 6d47abc0c651d60342661aad6ef653fe8d522a251ce335ece278d5f6dfa8b65f: Status 404 returned error can't find the container with id 6d47abc0c651d60342661aad6ef653fe8d522a251ce335ece278d5f6dfa8b65f
Jul 17 21:54:13.787581 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:13.787552    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:14.373712 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:14.373667    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-2g44p" event={"ID":"8f5ee100-6729-445b-ac7d-60d590c960cc","Type":"ContainerStarted","Data":"6d47abc0c651d60342661aad6ef653fe8d522a251ce335ece278d5f6dfa8b65f"}
Jul 17 21:54:16.385768 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:16.385735    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-2g44p" event={"ID":"8f5ee100-6729-445b-ac7d-60d590c960cc","Type":"ContainerStarted","Data":"c73e2f235b6ec1e08c693887bda1b05e01209b7bdadf0de1379d4fc4933a4cb5"}
Jul 17 21:54:16.386216 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:16.385839    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-2g44p"
Jul 17 21:54:16.409542 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:16.409439    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-2g44p" podStartSLOduration=0.992082918 podStartE2EDuration="3.409422574s" totalImagesPullingTime="2.417339656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:54:16.407936267 +0000 UTC m=+519.796952931" watchObservedRunningTime="2026-07-17 21:54:16.409422574 +0000 UTC m=+519.798439225"
Jul 17 21:54:27.392047 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:27.392009    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-2g44p"
Jul 17 21:54:29.773887 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:29.773797    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:29.910612 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:29.910582    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:30.285167 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:30.285130    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:42.573295 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:42.573261    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:42.673307 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:42.673272    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:49.197349 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:49.197307    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:49.280821 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:49.280783    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:49.341623 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:49.341592    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:49.529212 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:49.529156    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:52.964604 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:52.964564    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:54:53.065374 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:54:53.065338    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:02.237986 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:02.237951    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:12.652280 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:12.652244    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:18.384359 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:18.384323    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:18.484806 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:18.484760    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:24.710770 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:24.710734    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:24.810794 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:24.810758    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:25.366803 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:25.366772    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:25.548443 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:25.548411    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:25.854350 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:25.854309    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:25.954088 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:25.954019    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:26.012446 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:26.012412    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 21:55:33.743008 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:33.742912    2565 generic.go:356] "Generic (PLEG): container finished" podID="fc686a0b-25d0-4d64-bf27-ff5ee005090b" containerID="161f99e0b9cf6c53b4a73a6e5bf5d0e7595f484182983407f2dd29ad3a1f3175" exitCode=0
Jul 17 21:55:33.743008 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:33.742981    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2" event={"ID":"fc686a0b-25d0-4d64-bf27-ff5ee005090b","Type":"ContainerDied","Data":"161f99e0b9cf6c53b4a73a6e5bf5d0e7595f484182983407f2dd29ad3a1f3175"}
Jul 17 21:55:33.743499 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:33.743385    2565 scope.go:117] "RemoveContainer" containerID="161f99e0b9cf6c53b4a73a6e5bf5d0e7595f484182983407f2dd29ad3a1f3175"
Jul 17 21:55:33.744552 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:33.744518    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 21:55:33.744640 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:33.744611    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 21:55:33.744694 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:33.744680    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 21:55:34.749720 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:34.749682    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2" event={"ID":"fc686a0b-25d0-4d64-bf27-ff5ee005090b","Type":"ContainerStarted","Data":"1be962725876f60515402a349fc84e9b94ced040b8312504c5e87d5a3587b6d7"}
Jul 17 21:55:35.565444 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:35.564957    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:55:35.565444 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:35.565120    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-pvfz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Jul 17 21:55:35.565444 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:35.565167    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2" podUID="fc686a0b-25d0-4d64-bf27-ff5ee005090b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Jul 17 21:55:36.090297 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:36.090267    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:36.190573 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:36.190533    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:36.564319 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:36.564276    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-pvfz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Jul 17 21:55:36.564537 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:36.564345    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2" podUID="fc686a0b-25d0-4d64-bf27-ff5ee005090b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Jul 17 21:55:37.100333 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:37.100306    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 21:55:37.103279 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:37.103253    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 21:55:37.564639 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:37.564601    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-pvfz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Jul 17 21:55:37.564956 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:37.564922    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2" podUID="fc686a0b-25d0-4d64-bf27-ff5ee005090b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Jul 17 21:55:38.564443 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:38.564403    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-pvfz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Jul 17 21:55:38.565060 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:38.565022    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2" podUID="fc686a0b-25d0-4d64-bf27-ff5ee005090b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Jul 17 21:55:39.568443 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:39.568407    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:55:39.568905 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:39.568886    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:55:39.569671 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:39.569649    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pvfz2"
Jul 17 21:55:40.540768 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:40.540714    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:41.136439 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:41.136401    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 21:55:47.959098 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:47.959050    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:55:49.038434 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:49.038402    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 21:55:59.260544 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:55:59.260427    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:56:00.432908 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:00.432864    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 21:56:01.659952 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:01.659912    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:56:14.083136 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:14.083086    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:56:18.547213 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.547181    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"]
Jul 17 21:56:18.553540 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.553508    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.562544 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.562518    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7nlsc\""
Jul 17 21:56:18.562733 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.562521    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 21:56:18.562854 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.562520    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 21:56:18.562854 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.562539    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 17 21:56:18.587483 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.587451    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"]
Jul 17 21:56:18.587759 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.587736    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82ec16d0-bf9c-4708-be9b-86209944c32b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.587834 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.587783    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82ec16d0-bf9c-4708-be9b-86209944c32b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.587896 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.587838    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82ec16d0-bf9c-4708-be9b-86209944c32b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.587962 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.587902    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d4sn\" (UniqueName: \"kubernetes.io/projected/82ec16d0-bf9c-4708-be9b-86209944c32b-kube-api-access-7d4sn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.588019 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.587957    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ec16d0-bf9c-4708-be9b-86209944c32b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.588019 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.587986    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82ec16d0-bf9c-4708-be9b-86209944c32b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.624979 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.624928    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:56:18.688978 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.688937    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82ec16d0-bf9c-4708-be9b-86209944c32b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.688978 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.688986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82ec16d0-bf9c-4708-be9b-86209944c32b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.689253 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.689017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d4sn\" (UniqueName: \"kubernetes.io/projected/82ec16d0-bf9c-4708-be9b-86209944c32b-kube-api-access-7d4sn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.689253 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.689091    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ec16d0-bf9c-4708-be9b-86209944c32b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.689253 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.689119    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82ec16d0-bf9c-4708-be9b-86209944c32b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.689253 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.689177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82ec16d0-bf9c-4708-be9b-86209944c32b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.689450 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.689377    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82ec16d0-bf9c-4708-be9b-86209944c32b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.689509 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.689453    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82ec16d0-bf9c-4708-be9b-86209944c32b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.689612 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.689585    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82ec16d0-bf9c-4708-be9b-86209944c32b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.691442 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.691416    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82ec16d0-bf9c-4708-be9b-86209944c32b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.691683 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.691665    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82ec16d0-bf9c-4708-be9b-86209944c32b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.699855 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.699831    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d4sn\" (UniqueName: \"kubernetes.io/projected/82ec16d0-bf9c-4708-be9b-86209944c32b-kube-api-access-7d4sn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65\" (UID: \"82ec16d0-bf9c-4708-be9b-86209944c32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:18.866428 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:18.866327    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:19.009829 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:19.009761    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"]
Jul 17 21:56:19.012103 ip-10-0-129-11 kubenswrapper[2565]: W0717 21:56:19.012046    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ec16d0_bf9c_4708_be9b_86209944c32b.slice/crio-928e120af308ca0a675174891ca71c0636e77d2c98601a4cf917f7de693bffd3 WatchSource:0}: Error finding container 928e120af308ca0a675174891ca71c0636e77d2c98601a4cf917f7de693bffd3: Status 404 returned error can't find the container with id 928e120af308ca0a675174891ca71c0636e77d2c98601a4cf917f7de693bffd3
Jul 17 21:56:19.013446 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:19.013429    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:56:19.957378 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:19.957328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65" event={"ID":"82ec16d0-bf9c-4708-be9b-86209944c32b","Type":"ContainerStarted","Data":"928e120af308ca0a675174891ca71c0636e77d2c98601a4cf917f7de693bffd3"}
Jul 17 21:56:20.239362 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:20.239279    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 21:56:20.964525 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:20.964491    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65" event={"ID":"82ec16d0-bf9c-4708-be9b-86209944c32b","Type":"ContainerStarted","Data":"902bae6bea037109a66c1cf56938086b78ce0f252e8bf416cfd0b8726e64b21f"}
Jul 17 21:56:20.981698 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:20.981647    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65" podStartSLOduration=1.947362735 podStartE2EDuration="2.981632251s" totalImagesPullingTime="1.034269516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:56:20.980510971 +0000 UTC m=+644.369527627" watchObservedRunningTime="2026-07-17 21:56:20.981632251 +0000 UTC m=+644.370648902"
Jul 17 21:56:28.867509 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:28.867450    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:28.867509 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:28.867529    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:28.880249 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:28.880215    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:29.011353 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:29.011324    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65"
Jul 17 21:56:29.534424 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:29.534382    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:56:31.550833 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:31.550798    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 21:56:38.648429 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:38.648388    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:56:41.252537 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:41.252499    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 21:56:57.712966 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:56:57.712926    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:17.575534 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:57:17.575497    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:17.646238 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:57:17.646203    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:57.430908 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:57:57.430873    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:57.519803 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:57:57.519761    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:58.591248 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:57:58.591216    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:57:58.691227 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:57:58.691197    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 21:58:23.531402 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:58:23.531366    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 21:58:35.943501 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:58:35.943460    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 21:58:44.345806 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:58:44.345772    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 21:58:55.776565 ip-10-0-129-11 kubenswrapper[2565]: I0717 21:58:55.776526    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:00:00.145627 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:00.145590    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738760-4d9n7"]
Jul 17 22:00:00.149140 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:00.149124    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-4d9n7"
Jul 17 22:00:00.151665 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:00.151637    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:00:00.159793 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:00.159743    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738760-4d9n7"]
Jul 17 22:00:00.159793 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:00.159770    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-4d9n7"
Jul 17 22:00:00.295899 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:00.295875    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738760-4d9n7"]
Jul 17 22:00:00.957097 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:00.957038    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-4d9n7" event={"ID":"f84846b0-0ae6-448d-9670-1d066562faa9","Type":"ContainerStarted","Data":"d0a754935d9ce9fbc82d31e4d97fba97757df303d024e4623d6d19791b32289c"}
Jul 17 22:00:05.463702 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:05.463651    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:00:06.665031 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:06.664995    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:00:18.039250 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:18.039213    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-4d9n7" event={"ID":"f84846b0-0ae6-448d-9670-1d066562faa9","Type":"ContainerStarted","Data":"8419876a62c397f3680a1b8c2867e0223af94a7cf64435fa6bcf33ef1446a254"}
Jul 17 22:00:18.057387 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:18.057312    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-4d9n7" podStartSLOduration=1.307716178 podStartE2EDuration="18.057291461s" totalImagesPullingTime="16.749575283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:00:18.05546562 +0000 UTC m=+881.444482272" watchObservedRunningTime="2026-07-17 22:00:18.057291461 +0000 UTC m=+881.446308112"
Jul 17 22:00:19.045558 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:19.045523    2565 generic.go:356] "Generic (PLEG): container finished" podID="f84846b0-0ae6-448d-9670-1d066562faa9" containerID="8419876a62c397f3680a1b8c2867e0223af94a7cf64435fa6bcf33ef1446a254" exitCode=0
Jul 17 22:00:19.045966 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:19.045581    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-4d9n7" event={"ID":"f84846b0-0ae6-448d-9670-1d066562faa9","Type":"ContainerDied","Data":"8419876a62c397f3680a1b8c2867e0223af94a7cf64435fa6bcf33ef1446a254"}
Jul 17 22:00:20.188010 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:20.187974    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-4d9n7"
Jul 17 22:00:21.056570 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:21.056538    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-4d9n7"
Jul 17 22:00:21.056745 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:21.056574    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-4d9n7" event={"ID":"f84846b0-0ae6-448d-9670-1d066562faa9","Type":"ContainerDied","Data":"d0a754935d9ce9fbc82d31e4d97fba97757df303d024e4623d6d19791b32289c"}
Jul 17 22:00:21.056745 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:21.056618    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0a754935d9ce9fbc82d31e4d97fba97757df303d024e4623d6d19791b32289c"
Jul 17 22:00:29.251924 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:29.251890    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:00:37.130451 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:37.130420    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:00:37.132867 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:37.131410    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:00:44.238857 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:44.238817    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:00:49.357644 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:49.357606    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:00:49.449636 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:49.449588    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:00:49.552857 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:49.552812    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:00:55.319917 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:55.319875    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:00:56.523477 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:00:56.523441    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:01:01.761801 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:01:01.761764    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:01:01.862544 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:01:01.862516    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:01:15.348007 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:01:15.347973    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:01:18.833887 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:01:18.833852    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:01:28.747194 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:01:28.747160    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:01:41.141607 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:01:41.141565    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:01:46.656258 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:01:46.656214    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:01:46.741629 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:01:46.741596    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:01:47.555743 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:01:47.555700    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:01:58.843667 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:01:58.843587    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:02:08.840358 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:02:08.840318    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:02:21.635416 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:02:21.635364    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:02:31.552967 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:02:31.552929    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:02:55.740153 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:02:55.740118    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:03:07.637115 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:03:07.637061    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:03:28.645212 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:03:28.645118    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:03:39.244531 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:03:39.244497    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:03:57.132495 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:03:57.132457    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:04:06.941422 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:04:06.941388    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:04:16.439057 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:04:16.439019    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:04:26.144441 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:04:26.144400    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:04:35.439690 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:04:35.439648    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:05:37.157948 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:05:37.157812    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:05:37.161993 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:05:37.159012    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:05:50.528862 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:05:50.528810    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:06:09.080027 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:06:09.079973    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:06:42.144255 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:06:42.144221    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:06:52.638671 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:06:52.638627    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:07:03.049926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:07:03.049886    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:07:15.439568 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:07:15.439528    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:07:24.840049 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:07:24.840011    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:07:36.537290 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:07:36.537256    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:07:45.236149 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:07:45.236102    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:07:56.533199 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:07:56.533158    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:07:59.225804 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:07:59.225725    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:08:06.536164 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:08:06.536126    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:08:15.731961 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:08:15.731922    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:08:27.052631 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:08:27.052588    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:08:48.253142 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:08:48.253098    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:08:58.347882 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:08:58.347846    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:09:08.138723 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:09:08.138685    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:09:17.644974 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:09:17.644934    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:09:27.837958 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:09:27.837918    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:09:39.020295 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:09:39.020261    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:09:53.143917 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:09:53.143876    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:10:02.749828 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:10:02.749781    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:10:16.551510 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:10:16.551475    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:10:25.741315 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:10:25.741279    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:10:31.753656 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:10:31.753616    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:10:37.187544 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:10:37.187440    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:10:37.191469 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:10:37.189537    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:10:41.547842 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:10:41.547811    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:10:55.940979 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:10:55.940934    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:11:04.935446 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:11:04.935356    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:11:15.643192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:11:15.643155    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:11:26.738378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:11:26.738345    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:11:44.144874 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:11:44.144835    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:11:58.648807 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:11:58.648766    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:12:07.838533 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:07.838494    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:12:11.408697 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:11.408648    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:18.206641 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:18.206599    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:21.436684 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:21.436646    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:12:26.456522 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:26.456476    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:30.647590 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:30.647505    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:12:31.557357 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:31.557322    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:34.464111 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:34.464054    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:34.562589 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:34.562560    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:34.924219 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:34.924192    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:38.836797 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:38.836759    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:12:40.816366 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:40.816336    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:40.920234 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:40.920197    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:41.053017 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.052989    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"]
Jul 17 22:12:41.053370 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.053357    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f84846b0-0ae6-448d-9670-1d066562faa9" containerName="cleanup"
Jul 17 22:12:41.053415 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.053372    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f84846b0-0ae6-448d-9670-1d066562faa9" containerName="cleanup"
Jul 17 22:12:41.053448 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.053432    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f84846b0-0ae6-448d-9670-1d066562faa9" containerName="cleanup"
Jul 17 22:12:41.056480 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.056455    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.060054 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.060029    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7fb802f2-openshift-default-dockercfg-6cd7q\""
Jul 17 22:12:41.101848 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.101762    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"]
Jul 17 22:12:41.110376 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.110349    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:41.188650 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.188616    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:41.211896 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.211865    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-credential-socket\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.212033 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.211904    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88jqp\" (UniqueName: \"kubernetes.io/projected/3b60617e-7f8f-4a38-9bce-07277bee1db2-kube-api-access-88jqp\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.212033 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.211933    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-workload-socket\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.212033 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.211989    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-podinfo\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.212192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.212054    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-workload-certs\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.212192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.212102    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b60617e-7f8f-4a38-9bce-07277bee1db2-istiod-ca-cert\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.212192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.212119    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-envoy\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.212192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.212137    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-data\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.212192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.212168    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-token\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.313298 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.313253    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-data\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.313496 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.313309    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-token\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.313496 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.313350    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-credential-socket\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.313496 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.313382    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88jqp\" (UniqueName: \"kubernetes.io/projected/3b60617e-7f8f-4a38-9bce-07277bee1db2-kube-api-access-88jqp\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.313496 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.313444    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-workload-socket\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.313739 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.313712    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-data\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.313799 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.313740    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-credential-socket\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.313859 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.313811    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-workload-socket\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.313859 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.313828    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-podinfo\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.313934 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.313890    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-workload-certs\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.313934 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.313907    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b60617e-7f8f-4a38-9bce-07277bee1db2-istiod-ca-cert\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.313934 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.313929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-envoy\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.314287 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.314264    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-workload-certs\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.314529 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.314508    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b60617e-7f8f-4a38-9bce-07277bee1db2-istiod-ca-cert\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.316307 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.316288    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-podinfo\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.316385 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.316317    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-envoy\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.322362 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.322341    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88jqp\" (UniqueName: \"kubernetes.io/projected/3b60617e-7f8f-4a38-9bce-07277bee1db2-kube-api-access-88jqp\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.322461 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.322359    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-token\") pod \"e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") " pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.367976 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.367891    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:41.508551 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.508526    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"]
Jul 17 22:12:41.510573 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:12:41.510544    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b60617e_7f8f_4a38_9bce_07277bee1db2.slice/crio-db4439daf25e3989353bb08204d49707f1f6eddef70d0298bf2989407e7aa781 WatchSource:0}: Error finding container db4439daf25e3989353bb08204d49707f1f6eddef70d0298bf2989407e7aa781: Status 404 returned error can't find the container with id db4439daf25e3989353bb08204d49707f1f6eddef70d0298bf2989407e7aa781
Jul 17 22:12:41.512702 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.512670    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:12:41.512793 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.512751    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:12:41.512793 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:41.512782    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:12:42.229978 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:42.229937    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:42.328850 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:42.328805    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:42.473494 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:42.473378    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz" event={"ID":"3b60617e-7f8f-4a38-9bce-07277bee1db2","Type":"ContainerStarted","Data":"db866da2aa79424d22543d81c7d7537ddb67422afe167c9195622ad832ea5392"}
Jul 17 22:12:42.473494 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:42.473460    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz" event={"ID":"3b60617e-7f8f-4a38-9bce-07277bee1db2","Type":"ContainerStarted","Data":"db4439daf25e3989353bb08204d49707f1f6eddef70d0298bf2989407e7aa781"}
Jul 17 22:12:42.494113 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:42.494002    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:42.499540 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:42.499496    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz" podStartSLOduration=1.499481029 podStartE2EDuration="1.499481029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:12:42.499258746 +0000 UTC m=+1625.888275398" watchObservedRunningTime="2026-07-17 22:12:42.499481029 +0000 UTC m=+1625.888497679"
Jul 17 22:12:42.890985 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:42.890951    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:42.988741 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:42.988710    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:43.096152 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:43.096123    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:43.368869 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:43.368831    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:43.373793 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:43.373769    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:43.477794 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:43.477766    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:43.478626 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:43.478606    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:44.993230 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:44.993200    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:12:47.220085 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:47.220044    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5"]
Jul 17 22:12:47.223709 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:47.223692    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5"
Jul 17 22:12:47.226235 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:47.226206    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:12:47.226352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:47.226286    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:12:47.231031 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:47.230966    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5"]
Jul 17 22:12:47.272197 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:47.272158    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34aa061-47e6-43be-b348-4e9c4172d193-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5\" (UID: \"f34aa061-47e6-43be-b348-4e9c4172d193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5"
Jul 17 22:12:47.373716 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:47.373659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34aa061-47e6-43be-b348-4e9c4172d193-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5\" (UID: \"f34aa061-47e6-43be-b348-4e9c4172d193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5"
Jul 17 22:12:47.374351 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:47.374329    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34aa061-47e6-43be-b348-4e9c4172d193-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5\" (UID: \"f34aa061-47e6-43be-b348-4e9c4172d193\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5"
Jul 17 22:12:47.537440 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:47.537406    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5"
Jul 17 22:12:47.675359 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:47.675330    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5"]
Jul 17 22:12:47.677392 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:12:47.677368    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34aa061_47e6_43be_b348_4e9c4172d193.slice/crio-0796767b9fb682134a84ee5b0d73f13fe16fd9a48e675e9d4269bca74428dc0b WatchSource:0}: Error finding container 0796767b9fb682134a84ee5b0d73f13fe16fd9a48e675e9d4269bca74428dc0b: Status 404 returned error can't find the container with id 0796767b9fb682134a84ee5b0d73f13fe16fd9a48e675e9d4269bca74428dc0b
Jul 17 22:12:48.501639 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:48.501607    2565 generic.go:356] "Generic (PLEG): container finished" podID="f34aa061-47e6-43be-b348-4e9c4172d193" containerID="470f580a68ec4b1c53a2724438438d7fe781795ad4701f6e0da4cbeb722f39e0" exitCode=7
Jul 17 22:12:48.502047 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:48.501690    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5" event={"ID":"f34aa061-47e6-43be-b348-4e9c4172d193","Type":"ContainerDied","Data":"470f580a68ec4b1c53a2724438438d7fe781795ad4701f6e0da4cbeb722f39e0"}
Jul 17 22:12:48.502047 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:48.501743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5" event={"ID":"f34aa061-47e6-43be-b348-4e9c4172d193","Type":"ContainerStarted","Data":"0796767b9fb682134a84ee5b0d73f13fe16fd9a48e675e9d4269bca74428dc0b"}
Jul 17 22:12:48.502047 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:48.501865    2565 scope.go:117] "RemoveContainer" containerID="470f580a68ec4b1c53a2724438438d7fe781795ad4701f6e0da4cbeb722f39e0"
Jul 17 22:12:49.086247 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:12:49.086051    2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34aa061_47e6_43be_b348_4e9c4172d193.slice/crio-conmon-cbbf1a9ed47e8f7c9089270f888743bce8afb452a8729cb07024f620b2061ab4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34aa061_47e6_43be_b348_4e9c4172d193.slice/crio-cbbf1a9ed47e8f7c9089270f888743bce8afb452a8729cb07024f620b2061ab4.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 22:12:49.346969 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:49.346884    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:12:49.508874 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:49.508838    2565 generic.go:356] "Generic (PLEG): container finished" podID="f34aa061-47e6-43be-b348-4e9c4172d193" containerID="cbbf1a9ed47e8f7c9089270f888743bce8afb452a8729cb07024f620b2061ab4" exitCode=7
Jul 17 22:12:49.509409 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:49.508892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5" event={"ID":"f34aa061-47e6-43be-b348-4e9c4172d193","Type":"ContainerDied","Data":"cbbf1a9ed47e8f7c9089270f888743bce8afb452a8729cb07024f620b2061ab4"}
Jul 17 22:12:49.509409 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:49.508941    2565 scope.go:117] "RemoveContainer" containerID="470f580a68ec4b1c53a2724438438d7fe781795ad4701f6e0da4cbeb722f39e0"
Jul 17 22:12:49.509409 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:49.509151    2565 scope.go:117] "RemoveContainer" containerID="cbbf1a9ed47e8f7c9089270f888743bce8afb452a8729cb07024f620b2061ab4"
Jul 17 22:12:49.509571 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:12:49.509411    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5_odh-ai-gateway-infra(f34aa061-47e6-43be-b348-4e9c4172d193)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5" podUID="f34aa061-47e6-43be-b348-4e9c4172d193"
Jul 17 22:12:50.515485 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:50.515458    2565 scope.go:117] "RemoveContainer" containerID="cbbf1a9ed47e8f7c9089270f888743bce8afb452a8729cb07024f620b2061ab4"
Jul 17 22:12:50.515871 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:12:50.515660    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5_odh-ai-gateway-infra(f34aa061-47e6-43be-b348-4e9c4172d193)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5" podUID="f34aa061-47e6-43be-b348-4e9c4172d193"
Jul 17 22:12:57.548729 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:57.548648    2565 generic.go:356] "Generic (PLEG): container finished" podID="3b60617e-7f8f-4a38-9bce-07277bee1db2" containerID="db866da2aa79424d22543d81c7d7537ddb67422afe167c9195622ad832ea5392" exitCode=0
Jul 17 22:12:57.548729 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:57.548716    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz" event={"ID":"3b60617e-7f8f-4a38-9bce-07277bee1db2","Type":"ContainerDied","Data":"db866da2aa79424d22543d81c7d7537ddb67422afe167c9195622ad832ea5392"}
Jul 17 22:12:57.549196 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:57.549134    2565 scope.go:117] "RemoveContainer" containerID="db866da2aa79424d22543d81c7d7537ddb67422afe167c9195622ad832ea5392"
Jul 17 22:12:57.550185 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:57.550155    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:12:57.550303 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:57.550222    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:12:57.550303 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:57.550253    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:12:58.478570 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:58.478521    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:58.555427 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:58.555392    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz" event={"ID":"3b60617e-7f8f-4a38-9bce-07277bee1db2","Type":"ContainerStarted","Data":"33852645e09692a0019cc768230397fd3f381fa6ca731ef64c5d3e172f284732"}
Jul 17 22:12:59.370242 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:59.370198    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:12:59.370449 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:59.370366    2565 patch_prober.go:28] interesting pod/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 17 22:12:59.370449 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:12:59.370413    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz" podUID="3b60617e-7f8f-4a38-9bce-07277bee1db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 17 22:13:00.368458 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:00.368402    2565 patch_prober.go:28] interesting pod/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 17 22:13:00.368989 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:00.368486    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz" podUID="3b60617e-7f8f-4a38-9bce-07277bee1db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 17 22:13:01.368545 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:01.368506    2565 patch_prober.go:28] interesting pod/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 17 22:13:01.369297 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:01.369263    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz" podUID="3b60617e-7f8f-4a38-9bce-07277bee1db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 17 22:13:02.368963 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:02.368917    2565 patch_prober.go:28] interesting pod/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 17 22:13:02.369497 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:02.369000    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz" podUID="3b60617e-7f8f-4a38-9bce-07277bee1db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 17 22:13:03.373275 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:03.373244    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:13:03.373713 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:03.373508    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:13:03.374102 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:03.374085    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:13:05.177191 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:05.177159    2565 scope.go:117] "RemoveContainer" containerID="cbbf1a9ed47e8f7c9089270f888743bce8afb452a8729cb07024f620b2061ab4"
Jul 17 22:13:05.597637 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:05.597601    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5" event={"ID":"f34aa061-47e6-43be-b348-4e9c4172d193","Type":"ContainerStarted","Data":"6858bfd41b04aed7bb0c23f791d53fc9ecd0f800a240dfb559085ecc157adfbb"}
Jul 17 22:13:05.613729 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:05.613678    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5" podStartSLOduration=18.613646099 podStartE2EDuration="18.613646099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:13:05.612755492 +0000 UTC m=+1649.001772145" watchObservedRunningTime="2026-07-17 22:13:05.613646099 +0000 UTC m=+1649.002662750"
Jul 17 22:13:06.604641 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:06.604586    2565 generic.go:356] "Generic (PLEG): container finished" podID="f34aa061-47e6-43be-b348-4e9c4172d193" containerID="6858bfd41b04aed7bb0c23f791d53fc9ecd0f800a240dfb559085ecc157adfbb" exitCode=7
Jul 17 22:13:06.605134 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:06.604673    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5" event={"ID":"f34aa061-47e6-43be-b348-4e9c4172d193","Type":"ContainerDied","Data":"6858bfd41b04aed7bb0c23f791d53fc9ecd0f800a240dfb559085ecc157adfbb"}
Jul 17 22:13:06.605134 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:06.604745    2565 scope.go:117] "RemoveContainer" containerID="cbbf1a9ed47e8f7c9089270f888743bce8afb452a8729cb07024f620b2061ab4"
Jul 17 22:13:06.605134 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:06.605116    2565 scope.go:117] "RemoveContainer" containerID="6858bfd41b04aed7bb0c23f791d53fc9ecd0f800a240dfb559085ecc157adfbb"
Jul 17 22:13:06.605391 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:13:06.605368    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5_odh-ai-gateway-infra(f34aa061-47e6-43be-b348-4e9c4172d193)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5" podUID="f34aa061-47e6-43be-b348-4e9c4172d193"
Jul 17 22:13:06.629290 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:06.629252    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5"]
Jul 17 22:13:07.752655 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:07.752627    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5"
Jul 17 22:13:07.860019 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:07.859982    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34aa061-47e6-43be-b348-4e9c4172d193-maas-api-service-ca\") pod \"f34aa061-47e6-43be-b348-4e9c4172d193\" (UID: \"f34aa061-47e6-43be-b348-4e9c4172d193\") "
Jul 17 22:13:07.860375 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:07.860352    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34aa061-47e6-43be-b348-4e9c4172d193-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f34aa061-47e6-43be-b348-4e9c4172d193" (UID: "f34aa061-47e6-43be-b348-4e9c4172d193"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:13:07.961260 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:07.961173    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34aa061-47e6-43be-b348-4e9c4172d193-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:13:08.617035 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:08.617007    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5"
Jul 17 22:13:08.617244 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:08.617048    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5" event={"ID":"f34aa061-47e6-43be-b348-4e9c4172d193","Type":"ContainerDied","Data":"0796767b9fb682134a84ee5b0d73f13fe16fd9a48e675e9d4269bca74428dc0b"}
Jul 17 22:13:08.617244 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:08.617135    2565 scope.go:117] "RemoveContainer" containerID="6858bfd41b04aed7bb0c23f791d53fc9ecd0f800a240dfb559085ecc157adfbb"
Jul 17 22:13:08.639777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:08.639746    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5"]
Jul 17 22:13:08.643583 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:08.643556    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rwsd5"]
Jul 17 22:13:09.182540 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:09.182508    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f34aa061-47e6-43be-b348-4e9c4172d193" path="/var/lib/kubelet/pods/f34aa061-47e6-43be-b348-4e9c4172d193/volumes"
Jul 17 22:13:47.268766 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.268731    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq"]
Jul 17 22:13:47.269195 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.269118    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f34aa061-47e6-43be-b348-4e9c4172d193" containerName="revoke-keys"
Jul 17 22:13:47.269195 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.269129    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f34aa061-47e6-43be-b348-4e9c4172d193" containerName="revoke-keys"
Jul 17 22:13:47.269195 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.269136    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f34aa061-47e6-43be-b348-4e9c4172d193" containerName="revoke-keys"
Jul 17 22:13:47.269195 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.269142    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f34aa061-47e6-43be-b348-4e9c4172d193" containerName="revoke-keys"
Jul 17 22:13:47.269328 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.269209    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f34aa061-47e6-43be-b348-4e9c4172d193" containerName="revoke-keys"
Jul 17 22:13:47.269328 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.269217    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f34aa061-47e6-43be-b348-4e9c4172d193" containerName="revoke-keys"
Jul 17 22:13:47.269328 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.269225    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f34aa061-47e6-43be-b348-4e9c4172d193" containerName="revoke-keys"
Jul 17 22:13:47.272741 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.272724    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq"
Jul 17 22:13:47.275471 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.275450    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:13:47.275603 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.275466    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:13:47.280307 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.280260    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq"]
Jul 17 22:13:47.312046 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.312008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/038f7a0b-7fad-4367-9cff-be2db3d754eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq\" (UID: \"038f7a0b-7fad-4367-9cff-be2db3d754eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq"
Jul 17 22:13:47.413205 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.413155    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/038f7a0b-7fad-4367-9cff-be2db3d754eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq\" (UID: \"038f7a0b-7fad-4367-9cff-be2db3d754eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq"
Jul 17 22:13:47.413788 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.413766    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/038f7a0b-7fad-4367-9cff-be2db3d754eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq\" (UID: \"038f7a0b-7fad-4367-9cff-be2db3d754eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq"
Jul 17 22:13:47.584617 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.584533    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq"
Jul 17 22:13:47.722478 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.722452    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq"]
Jul 17 22:13:47.724727 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:13:47.724695    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038f7a0b_7fad_4367_9cff_be2db3d754eb.slice/crio-5657b1234a538dff2f3cec96499aa3db817b210d9caa7ce5b4d6f0617871c9d4 WatchSource:0}: Error finding container 5657b1234a538dff2f3cec96499aa3db817b210d9caa7ce5b4d6f0617871c9d4: Status 404 returned error can't find the container with id 5657b1234a538dff2f3cec96499aa3db817b210d9caa7ce5b4d6f0617871c9d4
Jul 17 22:13:47.798182 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:47.798140    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq" event={"ID":"038f7a0b-7fad-4367-9cff-be2db3d754eb","Type":"ContainerStarted","Data":"5657b1234a538dff2f3cec96499aa3db817b210d9caa7ce5b4d6f0617871c9d4"}
Jul 17 22:13:48.803982 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:48.803949    2565 generic.go:356] "Generic (PLEG): container finished" podID="038f7a0b-7fad-4367-9cff-be2db3d754eb" containerID="42a33dc03823d75674f857c6d82400af2bdda448b8681874b2cab021a67b7b20" exitCode=7
Jul 17 22:13:48.804405 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:48.804044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq" event={"ID":"038f7a0b-7fad-4367-9cff-be2db3d754eb","Type":"ContainerDied","Data":"42a33dc03823d75674f857c6d82400af2bdda448b8681874b2cab021a67b7b20"}
Jul 17 22:13:48.804405 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:48.804244    2565 scope.go:117] "RemoveContainer" containerID="42a33dc03823d75674f857c6d82400af2bdda448b8681874b2cab021a67b7b20"
Jul 17 22:13:49.809974 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:49.809936    2565 generic.go:356] "Generic (PLEG): container finished" podID="038f7a0b-7fad-4367-9cff-be2db3d754eb" containerID="4b9a267f4671b1868e15780d9f130d7804d2008eb4209e41f84708c085327a31" exitCode=7
Jul 17 22:13:49.810398 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:49.810033    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq" event={"ID":"038f7a0b-7fad-4367-9cff-be2db3d754eb","Type":"ContainerDied","Data":"4b9a267f4671b1868e15780d9f130d7804d2008eb4209e41f84708c085327a31"}
Jul 17 22:13:49.810398 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:49.810112    2565 scope.go:117] "RemoveContainer" containerID="42a33dc03823d75674f857c6d82400af2bdda448b8681874b2cab021a67b7b20"
Jul 17 22:13:49.810398 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:49.810322    2565 scope.go:117] "RemoveContainer" containerID="4b9a267f4671b1868e15780d9f130d7804d2008eb4209e41f84708c085327a31"
Jul 17 22:13:49.810585 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:13:49.810566    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq_odh-ai-gateway-infra(038f7a0b-7fad-4367-9cff-be2db3d754eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq" podUID="038f7a0b-7fad-4367-9cff-be2db3d754eb"
Jul 17 22:13:50.817158 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:13:50.817125    2565 scope.go:117] "RemoveContainer" containerID="4b9a267f4671b1868e15780d9f130d7804d2008eb4209e41f84708c085327a31"
Jul 17 22:13:50.817585 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:13:50.817347    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq_odh-ai-gateway-infra(038f7a0b-7fad-4367-9cff-be2db3d754eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq" podUID="038f7a0b-7fad-4367-9cff-be2db3d754eb"
Jul 17 22:14:04.176783 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:04.176692    2565 scope.go:117] "RemoveContainer" containerID="4b9a267f4671b1868e15780d9f130d7804d2008eb4209e41f84708c085327a31"
Jul 17 22:14:04.887584 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:04.887554    2565 generic.go:356] "Generic (PLEG): container finished" podID="038f7a0b-7fad-4367-9cff-be2db3d754eb" containerID="3838ab8c9cd7781fe74cdeceada5837e71aa418a53bf5ea4fb208d1a86307939" exitCode=7
Jul 17 22:14:04.887756 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:04.887628    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq" event={"ID":"038f7a0b-7fad-4367-9cff-be2db3d754eb","Type":"ContainerDied","Data":"3838ab8c9cd7781fe74cdeceada5837e71aa418a53bf5ea4fb208d1a86307939"}
Jul 17 22:14:04.887756 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:04.887696    2565 scope.go:117] "RemoveContainer" containerID="4b9a267f4671b1868e15780d9f130d7804d2008eb4209e41f84708c085327a31"
Jul 17 22:14:04.888097 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:04.888060    2565 scope.go:117] "RemoveContainer" containerID="3838ab8c9cd7781fe74cdeceada5837e71aa418a53bf5ea4fb208d1a86307939"
Jul 17 22:14:04.888359 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:14:04.888340    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq_odh-ai-gateway-infra(038f7a0b-7fad-4367-9cff-be2db3d754eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq" podUID="038f7a0b-7fad-4367-9cff-be2db3d754eb"
Jul 17 22:14:05.912473 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:05.912441    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq"]
Jul 17 22:14:06.051060 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:06.051032    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq"
Jul 17 22:14:06.194033 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:06.193930    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/038f7a0b-7fad-4367-9cff-be2db3d754eb-maas-api-service-ca\") pod \"038f7a0b-7fad-4367-9cff-be2db3d754eb\" (UID: \"038f7a0b-7fad-4367-9cff-be2db3d754eb\") "
Jul 17 22:14:06.194346 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:06.194325    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/038f7a0b-7fad-4367-9cff-be2db3d754eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "038f7a0b-7fad-4367-9cff-be2db3d754eb" (UID: "038f7a0b-7fad-4367-9cff-be2db3d754eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:14:06.295481 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:06.295448    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/038f7a0b-7fad-4367-9cff-be2db3d754eb-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:14:06.903159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:06.903128    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq"
Jul 17 22:14:06.903341 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:06.903157    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq" event={"ID":"038f7a0b-7fad-4367-9cff-be2db3d754eb","Type":"ContainerDied","Data":"5657b1234a538dff2f3cec96499aa3db817b210d9caa7ce5b4d6f0617871c9d4"}
Jul 17 22:14:06.903341 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:06.903217    2565 scope.go:117] "RemoveContainer" containerID="3838ab8c9cd7781fe74cdeceada5837e71aa418a53bf5ea4fb208d1a86307939"
Jul 17 22:14:06.925418 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:06.925387    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq"]
Jul 17 22:14:06.931241 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:06.931218    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-fmhlq"]
Jul 17 22:14:07.184168 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:07.184046    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="038f7a0b-7fad-4367-9cff-be2db3d754eb" path="/var/lib/kubelet/pods/038f7a0b-7fad-4367-9cff-be2db3d754eb/volumes"
Jul 17 22:14:26.573586 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:26.573549    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:14:37.345378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.345342    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z"]
Jul 17 22:14:37.345752 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.345695    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="038f7a0b-7fad-4367-9cff-be2db3d754eb" containerName="revoke-keys"
Jul 17 22:14:37.345752 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.345706    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="038f7a0b-7fad-4367-9cff-be2db3d754eb" containerName="revoke-keys"
Jul 17 22:14:37.345752 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.345714    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="038f7a0b-7fad-4367-9cff-be2db3d754eb" containerName="revoke-keys"
Jul 17 22:14:37.345752 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.345719    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="038f7a0b-7fad-4367-9cff-be2db3d754eb" containerName="revoke-keys"
Jul 17 22:14:37.345752 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.345732    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f34aa061-47e6-43be-b348-4e9c4172d193" containerName="revoke-keys"
Jul 17 22:14:37.345752 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.345738    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f34aa061-47e6-43be-b348-4e9c4172d193" containerName="revoke-keys"
Jul 17 22:14:37.345935 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.345800    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="038f7a0b-7fad-4367-9cff-be2db3d754eb" containerName="revoke-keys"
Jul 17 22:14:37.345935 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.345808    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="038f7a0b-7fad-4367-9cff-be2db3d754eb" containerName="revoke-keys"
Jul 17 22:14:37.349099 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.349057    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z"
Jul 17 22:14:37.354197 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.354172    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:14:37.354342 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.354173    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:14:37.368836 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.368803    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z"]
Jul 17 22:14:37.488483 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.488449    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5adbaf-cd04-468d-ae94-0a839bf40b58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z\" (UID: \"9f5adbaf-cd04-468d-ae94-0a839bf40b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z"
Jul 17 22:14:37.589779 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.589742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5adbaf-cd04-468d-ae94-0a839bf40b58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z\" (UID: \"9f5adbaf-cd04-468d-ae94-0a839bf40b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z"
Jul 17 22:14:37.590365 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.590347    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5adbaf-cd04-468d-ae94-0a839bf40b58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z\" (UID: \"9f5adbaf-cd04-468d-ae94-0a839bf40b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z"
Jul 17 22:14:37.659793 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.659737    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z"
Jul 17 22:14:37.795230 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:37.795208    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z"]
Jul 17 22:14:37.797517 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:14:37.797486    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5adbaf_cd04_468d_ae94_0a839bf40b58.slice/crio-5f16f91418c4b50e73584dfa44d2ade24735e4fa505ab01280217ff121187151 WatchSource:0}: Error finding container 5f16f91418c4b50e73584dfa44d2ade24735e4fa505ab01280217ff121187151: Status 404 returned error can't find the container with id 5f16f91418c4b50e73584dfa44d2ade24735e4fa505ab01280217ff121187151
Jul 17 22:14:38.045307 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:38.045267    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z" event={"ID":"9f5adbaf-cd04-468d-ae94-0a839bf40b58","Type":"ContainerStarted","Data":"5f16f91418c4b50e73584dfa44d2ade24735e4fa505ab01280217ff121187151"}
Jul 17 22:14:39.050989 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:39.050955    2565 generic.go:356] "Generic (PLEG): container finished" podID="9f5adbaf-cd04-468d-ae94-0a839bf40b58" containerID="8f7738a9f5d41018967982e78ff482d8c456b24ed81d764e98d14b9045935394" exitCode=7
Jul 17 22:14:39.051422 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:39.051047    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z" event={"ID":"9f5adbaf-cd04-468d-ae94-0a839bf40b58","Type":"ContainerDied","Data":"8f7738a9f5d41018967982e78ff482d8c456b24ed81d764e98d14b9045935394"}
Jul 17 22:14:39.051422 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:39.051243    2565 scope.go:117] "RemoveContainer" containerID="8f7738a9f5d41018967982e78ff482d8c456b24ed81d764e98d14b9045935394"
Jul 17 22:14:40.057372 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:40.057271    2565 generic.go:356] "Generic (PLEG): container finished" podID="9f5adbaf-cd04-468d-ae94-0a839bf40b58" containerID="5dad5494dd45b67f78350bfbbe9ea924a28485cf8d8e975d2229a954bf8ca424" exitCode=7
Jul 17 22:14:40.057833 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:40.057359    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z" event={"ID":"9f5adbaf-cd04-468d-ae94-0a839bf40b58","Type":"ContainerDied","Data":"5dad5494dd45b67f78350bfbbe9ea924a28485cf8d8e975d2229a954bf8ca424"}
Jul 17 22:14:40.057833 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:40.057427    2565 scope.go:117] "RemoveContainer" containerID="8f7738a9f5d41018967982e78ff482d8c456b24ed81d764e98d14b9045935394"
Jul 17 22:14:40.057833 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:40.057553    2565 scope.go:117] "RemoveContainer" containerID="5dad5494dd45b67f78350bfbbe9ea924a28485cf8d8e975d2229a954bf8ca424"
Jul 17 22:14:40.057833 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:14:40.057776    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z_odh-ai-gateway-infra(9f5adbaf-cd04-468d-ae94-0a839bf40b58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z" podUID="9f5adbaf-cd04-468d-ae94-0a839bf40b58"
Jul 17 22:14:41.063747 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:41.063719    2565 scope.go:117] "RemoveContainer" containerID="5dad5494dd45b67f78350bfbbe9ea924a28485cf8d8e975d2229a954bf8ca424"
Jul 17 22:14:41.064184 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:14:41.063935    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z_odh-ai-gateway-infra(9f5adbaf-cd04-468d-ae94-0a839bf40b58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z" podUID="9f5adbaf-cd04-468d-ae94-0a839bf40b58"
Jul 17 22:14:48.867325 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:48.867282    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:14:48.968214 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:48.968177    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:14:53.176819 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:53.176786    2565 scope.go:117] "RemoveContainer" containerID="5dad5494dd45b67f78350bfbbe9ea924a28485cf8d8e975d2229a954bf8ca424"
Jul 17 22:14:54.129371 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:54.129285    2565 generic.go:356] "Generic (PLEG): container finished" podID="9f5adbaf-cd04-468d-ae94-0a839bf40b58" containerID="44c07560bd936e2248a8e88b261ac8044b1536f2fc8d93afe5278798663c3189" exitCode=7
Jul 17 22:14:54.129371 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:54.129356    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z" event={"ID":"9f5adbaf-cd04-468d-ae94-0a839bf40b58","Type":"ContainerDied","Data":"44c07560bd936e2248a8e88b261ac8044b1536f2fc8d93afe5278798663c3189"}
Jul 17 22:14:54.129585 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:54.129417    2565 scope.go:117] "RemoveContainer" containerID="5dad5494dd45b67f78350bfbbe9ea924a28485cf8d8e975d2229a954bf8ca424"
Jul 17 22:14:54.129764 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:54.129747    2565 scope.go:117] "RemoveContainer" containerID="44c07560bd936e2248a8e88b261ac8044b1536f2fc8d93afe5278798663c3189"
Jul 17 22:14:54.130025 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:14:54.130003    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z_odh-ai-gateway-infra(9f5adbaf-cd04-468d-ae94-0a839bf40b58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z" podUID="9f5adbaf-cd04-468d-ae94-0a839bf40b58"
Jul 17 22:14:55.156811 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:55.156773    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z"]
Jul 17 22:14:55.292037 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:55.292009    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z"
Jul 17 22:14:55.356909 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:55.356875    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5adbaf-cd04-468d-ae94-0a839bf40b58-maas-api-service-ca\") pod \"9f5adbaf-cd04-468d-ae94-0a839bf40b58\" (UID: \"9f5adbaf-cd04-468d-ae94-0a839bf40b58\") "
Jul 17 22:14:55.357248 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:55.357227    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5adbaf-cd04-468d-ae94-0a839bf40b58-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f5adbaf-cd04-468d-ae94-0a839bf40b58" (UID: "9f5adbaf-cd04-468d-ae94-0a839bf40b58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:14:55.457607 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:55.457520    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5adbaf-cd04-468d-ae94-0a839bf40b58-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:14:56.141666 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:56.141639    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z"
Jul 17 22:14:56.141666 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:56.141656    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z" event={"ID":"9f5adbaf-cd04-468d-ae94-0a839bf40b58","Type":"ContainerDied","Data":"5f16f91418c4b50e73584dfa44d2ade24735e4fa505ab01280217ff121187151"}
Jul 17 22:14:56.141919 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:56.141721    2565 scope.go:117] "RemoveContainer" containerID="44c07560bd936e2248a8e88b261ac8044b1536f2fc8d93afe5278798663c3189"
Jul 17 22:14:56.172465 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:56.172431    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z"]
Jul 17 22:14:56.176500 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:56.176472    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-87j8z"]
Jul 17 22:14:57.182816 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:14:57.182780    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f5adbaf-cd04-468d-ae94-0a839bf40b58" path="/var/lib/kubelet/pods/9f5adbaf-cd04-468d-ae94-0a839bf40b58/volumes"
Jul 17 22:15:00.141751 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.141716    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m"]
Jul 17 22:15:00.142378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.142056    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f5adbaf-cd04-468d-ae94-0a839bf40b58" containerName="revoke-keys"
Jul 17 22:15:00.142378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.142096    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f5adbaf-cd04-468d-ae94-0a839bf40b58" containerName="revoke-keys"
Jul 17 22:15:00.142378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.142105    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f5adbaf-cd04-468d-ae94-0a839bf40b58" containerName="revoke-keys"
Jul 17 22:15:00.142378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.142111    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f5adbaf-cd04-468d-ae94-0a839bf40b58" containerName="revoke-keys"
Jul 17 22:15:00.142378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.142135    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="038f7a0b-7fad-4367-9cff-be2db3d754eb" containerName="revoke-keys"
Jul 17 22:15:00.142378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.142141    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="038f7a0b-7fad-4367-9cff-be2db3d754eb" containerName="revoke-keys"
Jul 17 22:15:00.142378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.142188    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f5adbaf-cd04-468d-ae94-0a839bf40b58" containerName="revoke-keys"
Jul 17 22:15:00.142378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.142194    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="038f7a0b-7fad-4367-9cff-be2db3d754eb" containerName="revoke-keys"
Jul 17 22:15:00.142378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.142202    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f5adbaf-cd04-468d-ae94-0a839bf40b58" containerName="revoke-keys"
Jul 17 22:15:00.142378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.142209    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f5adbaf-cd04-468d-ae94-0a839bf40b58" containerName="revoke-keys"
Jul 17 22:15:00.146499 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.146478    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738775-lbrm2"]
Jul 17 22:15:00.146643 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.146619    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m"
Jul 17 22:15:00.146833 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.146819    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f5adbaf-cd04-468d-ae94-0a839bf40b58" containerName="revoke-keys"
Jul 17 22:15:00.146833 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.146833    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f5adbaf-cd04-468d-ae94-0a839bf40b58" containerName="revoke-keys"
Jul 17 22:15:00.149007 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.148988    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:15:00.149824 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.149802    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-lbrm2"
Jul 17 22:15:00.152221 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.152197    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738775-lbrm2"]
Jul 17 22:15:00.155756 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.155733    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m"]
Jul 17 22:15:00.160135 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.160104    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m"
Jul 17 22:15:00.167884 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.167862    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-lbrm2"
Jul 17 22:15:00.314470 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.314442    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m"]
Jul 17 22:15:00.316516 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:15:00.316487    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586d9afe_03d6_4e2d_a04f_6e1ca465abbd.slice/crio-2c2bcc3454e62b68fba565108d4fe338debf5996bc839a3cc25dd3740cf872a7 WatchSource:0}: Error finding container 2c2bcc3454e62b68fba565108d4fe338debf5996bc839a3cc25dd3740cf872a7: Status 404 returned error can't find the container with id 2c2bcc3454e62b68fba565108d4fe338debf5996bc839a3cc25dd3740cf872a7
Jul 17 22:15:00.343576 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:00.343545    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738775-lbrm2"]
Jul 17 22:15:01.169726 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:01.169690    2565 generic.go:356] "Generic (PLEG): container finished" podID="586d9afe-03d6-4e2d-a04f-6e1ca465abbd" containerID="1dcb67e4e9520385c148a74741cb3621b38b7d7ccd4f468c155fa215a14ef426" exitCode=7
Jul 17 22:15:01.170176 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:01.169766    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m" event={"ID":"586d9afe-03d6-4e2d-a04f-6e1ca465abbd","Type":"ContainerDied","Data":"1dcb67e4e9520385c148a74741cb3621b38b7d7ccd4f468c155fa215a14ef426"}
Jul 17 22:15:01.170176 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:01.169828    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m" event={"ID":"586d9afe-03d6-4e2d-a04f-6e1ca465abbd","Type":"ContainerStarted","Data":"2c2bcc3454e62b68fba565108d4fe338debf5996bc839a3cc25dd3740cf872a7"}
Jul 17 22:15:01.170176 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:01.169902    2565 scope.go:117] "RemoveContainer" containerID="1dcb67e4e9520385c148a74741cb3621b38b7d7ccd4f468c155fa215a14ef426"
Jul 17 22:15:01.171302 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:01.171272    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-lbrm2" event={"ID":"053efeeb-991e-4e1d-9146-4e3872cb90c8","Type":"ContainerStarted","Data":"79cbd95df5288fe0fa74cc3fe9a7db2bc6ddd44dc64ade3a1c119a4b553a1e3d"}
Jul 17 22:15:01.171412 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:01.171312    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-lbrm2" event={"ID":"053efeeb-991e-4e1d-9146-4e3872cb90c8","Type":"ContainerStarted","Data":"b652a9fd9435b76af09e9764ee05fc8ae673560a38f93dfc66eb57c323bb3fba"}
Jul 17 22:15:01.200965 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:01.200916    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-lbrm2" podStartSLOduration=1.200900758 podStartE2EDuration="1.200900758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:15:01.199894614 +0000 UTC m=+1764.588911264" watchObservedRunningTime="2026-07-17 22:15:01.200900758 +0000 UTC m=+1764.589917408"
Jul 17 22:15:02.180551 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:02.180459    2565 generic.go:356] "Generic (PLEG): container finished" podID="053efeeb-991e-4e1d-9146-4e3872cb90c8" containerID="79cbd95df5288fe0fa74cc3fe9a7db2bc6ddd44dc64ade3a1c119a4b553a1e3d" exitCode=0
Jul 17 22:15:02.180996 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:02.180547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-lbrm2" event={"ID":"053efeeb-991e-4e1d-9146-4e3872cb90c8","Type":"ContainerDied","Data":"79cbd95df5288fe0fa74cc3fe9a7db2bc6ddd44dc64ade3a1c119a4b553a1e3d"}
Jul 17 22:15:02.182204 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:02.182179    2565 generic.go:356] "Generic (PLEG): container finished" podID="586d9afe-03d6-4e2d-a04f-6e1ca465abbd" containerID="3884b1fa9dcf81a8cd6ba5aa3cc4d60042e4e366c73015c055d682846f49c1a5" exitCode=7
Jul 17 22:15:02.182337 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:02.182220    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m" event={"ID":"586d9afe-03d6-4e2d-a04f-6e1ca465abbd","Type":"ContainerDied","Data":"3884b1fa9dcf81a8cd6ba5aa3cc4d60042e4e366c73015c055d682846f49c1a5"}
Jul 17 22:15:02.182337 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:02.182260    2565 scope.go:117] "RemoveContainer" containerID="1dcb67e4e9520385c148a74741cb3621b38b7d7ccd4f468c155fa215a14ef426"
Jul 17 22:15:02.182447 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:02.182435    2565 scope.go:117] "RemoveContainer" containerID="3884b1fa9dcf81a8cd6ba5aa3cc4d60042e4e366c73015c055d682846f49c1a5"
Jul 17 22:15:02.182681 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:15:02.182658    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m_odh-ai-gateway-infra(586d9afe-03d6-4e2d-a04f-6e1ca465abbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m" podUID="586d9afe-03d6-4e2d-a04f-6e1ca465abbd"
Jul 17 22:15:03.187866 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:03.187842    2565 scope.go:117] "RemoveContainer" containerID="3884b1fa9dcf81a8cd6ba5aa3cc4d60042e4e366c73015c055d682846f49c1a5"
Jul 17 22:15:03.188299 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:15:03.188060    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m_odh-ai-gateway-infra(586d9afe-03d6-4e2d-a04f-6e1ca465abbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m" podUID="586d9afe-03d6-4e2d-a04f-6e1ca465abbd"
Jul 17 22:15:03.322754 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:03.322733    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-lbrm2"
Jul 17 22:15:04.193394 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:04.193356    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-lbrm2"
Jul 17 22:15:04.193394 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:04.193391    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-lbrm2" event={"ID":"053efeeb-991e-4e1d-9146-4e3872cb90c8","Type":"ContainerDied","Data":"b652a9fd9435b76af09e9764ee05fc8ae673560a38f93dfc66eb57c323bb3fba"}
Jul 17 22:15:04.193941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:04.193436    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b652a9fd9435b76af09e9764ee05fc8ae673560a38f93dfc66eb57c323bb3fba"
Jul 17 22:15:14.176823 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:14.176785    2565 scope.go:117] "RemoveContainer" containerID="3884b1fa9dcf81a8cd6ba5aa3cc4d60042e4e366c73015c055d682846f49c1a5"
Jul 17 22:15:15.248323 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:15.248285    2565 generic.go:356] "Generic (PLEG): container finished" podID="586d9afe-03d6-4e2d-a04f-6e1ca465abbd" containerID="5bb18c5593763cee99f65ad65b8b125cd6f42644c8f259f1ea50f229653577e7" exitCode=7
Jul 17 22:15:15.248323 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:15.248322    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m" event={"ID":"586d9afe-03d6-4e2d-a04f-6e1ca465abbd","Type":"ContainerDied","Data":"5bb18c5593763cee99f65ad65b8b125cd6f42644c8f259f1ea50f229653577e7"}
Jul 17 22:15:15.248787 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:15.248369    2565 scope.go:117] "RemoveContainer" containerID="3884b1fa9dcf81a8cd6ba5aa3cc4d60042e4e366c73015c055d682846f49c1a5"
Jul 17 22:15:15.248787 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:15.248607    2565 scope.go:117] "RemoveContainer" containerID="5bb18c5593763cee99f65ad65b8b125cd6f42644c8f259f1ea50f229653577e7"
Jul 17 22:15:15.248880 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:15:15.248855    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m_odh-ai-gateway-infra(586d9afe-03d6-4e2d-a04f-6e1ca465abbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m" podUID="586d9afe-03d6-4e2d-a04f-6e1ca465abbd"
Jul 17 22:15:16.276521 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:16.276484    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m"]
Jul 17 22:15:16.414141 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:16.414113    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m"
Jul 17 22:15:17.261628 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:17.261589    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m"
Jul 17 22:15:17.261828 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:17.261591    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m" event={"ID":"586d9afe-03d6-4e2d-a04f-6e1ca465abbd","Type":"ContainerDied","Data":"2c2bcc3454e62b68fba565108d4fe338debf5996bc839a3cc25dd3740cf872a7"}
Jul 17 22:15:17.261828 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:17.261746    2565 scope.go:117] "RemoveContainer" containerID="5bb18c5593763cee99f65ad65b8b125cd6f42644c8f259f1ea50f229653577e7"
Jul 17 22:15:17.279794 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:17.279767    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m"]
Jul 17 22:15:17.285468 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:17.285441    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7fb802f2-29738775-v9d4m"]
Jul 17 22:15:19.182360 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:19.182325    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="586d9afe-03d6-4e2d-a04f-6e1ca465abbd" path="/var/lib/kubelet/pods/586d9afe-03d6-4e2d-a04f-6e1ca465abbd/volumes"
Jul 17 22:15:27.367773 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.367736    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87"]
Jul 17 22:15:27.368230 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.368138    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="586d9afe-03d6-4e2d-a04f-6e1ca465abbd" containerName="cleanup"
Jul 17 22:15:27.368230 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.368152    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="586d9afe-03d6-4e2d-a04f-6e1ca465abbd" containerName="cleanup"
Jul 17 22:15:27.368230 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.368166    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="586d9afe-03d6-4e2d-a04f-6e1ca465abbd" containerName="cleanup"
Jul 17 22:15:27.368230 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.368171    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="586d9afe-03d6-4e2d-a04f-6e1ca465abbd" containerName="cleanup"
Jul 17 22:15:27.368230 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.368182    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="053efeeb-991e-4e1d-9146-4e3872cb90c8" containerName="cleanup"
Jul 17 22:15:27.368230 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.368187    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="053efeeb-991e-4e1d-9146-4e3872cb90c8" containerName="cleanup"
Jul 17 22:15:27.368230 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.368204    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="586d9afe-03d6-4e2d-a04f-6e1ca465abbd" containerName="cleanup"
Jul 17 22:15:27.368230 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.368212    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="586d9afe-03d6-4e2d-a04f-6e1ca465abbd" containerName="cleanup"
Jul 17 22:15:27.368478 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.368272    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="586d9afe-03d6-4e2d-a04f-6e1ca465abbd" containerName="cleanup"
Jul 17 22:15:27.368478 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.368281    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="053efeeb-991e-4e1d-9146-4e3872cb90c8" containerName="cleanup"
Jul 17 22:15:27.368478 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.368289    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="586d9afe-03d6-4e2d-a04f-6e1ca465abbd" containerName="cleanup"
Jul 17 22:15:27.368478 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.368298    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="586d9afe-03d6-4e2d-a04f-6e1ca465abbd" containerName="cleanup"
Jul 17 22:15:27.372645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.372622    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87"
Jul 17 22:15:27.375346 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.375323    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:15:27.375473 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.375416    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:15:27.379446 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.379420    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87"]
Jul 17 22:15:27.454922 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.454882    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e339252c-d34f-4029-9619-68321503eea2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87\" (UID: \"e339252c-d34f-4029-9619-68321503eea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87"
Jul 17 22:15:27.555528 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.555490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e339252c-d34f-4029-9619-68321503eea2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87\" (UID: \"e339252c-d34f-4029-9619-68321503eea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87"
Jul 17 22:15:27.556163 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.556128    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e339252c-d34f-4029-9619-68321503eea2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87\" (UID: \"e339252c-d34f-4029-9619-68321503eea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87"
Jul 17 22:15:27.685336 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.685232    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87"
Jul 17 22:15:27.820526 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:27.820493    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87"]
Jul 17 22:15:27.823552 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:15:27.823513    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode339252c_d34f_4029_9619_68321503eea2.slice/crio-be511329b43a1269ec015ed77dc9c0307952d008b5e2264b571dd8e6668660f5 WatchSource:0}: Error finding container be511329b43a1269ec015ed77dc9c0307952d008b5e2264b571dd8e6668660f5: Status 404 returned error can't find the container with id be511329b43a1269ec015ed77dc9c0307952d008b5e2264b571dd8e6668660f5
Jul 17 22:15:28.318175 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:28.318136    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87" event={"ID":"e339252c-d34f-4029-9619-68321503eea2","Type":"ContainerStarted","Data":"ee882cac4b5f899d1ada4b3073cf31877f91005ded8825486d906fca584f9ecb"}
Jul 17 22:15:28.318359 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:28.318204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87" event={"ID":"e339252c-d34f-4029-9619-68321503eea2","Type":"ContainerStarted","Data":"be511329b43a1269ec015ed77dc9c0307952d008b5e2264b571dd8e6668660f5"}
Jul 17 22:15:28.334835 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:28.334725    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87" podStartSLOduration=1.3347068420000001 podStartE2EDuration="1.334706842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:15:28.332689459 +0000 UTC m=+1791.721706110" watchObservedRunningTime="2026-07-17 22:15:28.334706842 +0000 UTC m=+1791.723723492"
Jul 17 22:15:28.784470 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:28.784402    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:15:28.885007 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:28.884957    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:15:29.324358 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:29.324319    2565 generic.go:356] "Generic (PLEG): container finished" podID="e339252c-d34f-4029-9619-68321503eea2" containerID="ee882cac4b5f899d1ada4b3073cf31877f91005ded8825486d906fca584f9ecb" exitCode=7
Jul 17 22:15:29.324546 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:29.324371    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87" event={"ID":"e339252c-d34f-4029-9619-68321503eea2","Type":"ContainerDied","Data":"ee882cac4b5f899d1ada4b3073cf31877f91005ded8825486d906fca584f9ecb"}
Jul 17 22:15:29.324601 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:29.324589    2565 scope.go:117] "RemoveContainer" containerID="ee882cac4b5f899d1ada4b3073cf31877f91005ded8825486d906fca584f9ecb"
Jul 17 22:15:30.330851 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:30.330809    2565 generic.go:356] "Generic (PLEG): container finished" podID="e339252c-d34f-4029-9619-68321503eea2" containerID="7ae3e1b2f0052a82da46a298bd0a5188796d6579ac8b7d3d0bbd11f51358d760" exitCode=7
Jul 17 22:15:30.331356 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:30.330898    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87" event={"ID":"e339252c-d34f-4029-9619-68321503eea2","Type":"ContainerDied","Data":"7ae3e1b2f0052a82da46a298bd0a5188796d6579ac8b7d3d0bbd11f51358d760"}
Jul 17 22:15:30.331356 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:30.330960    2565 scope.go:117] "RemoveContainer" containerID="ee882cac4b5f899d1ada4b3073cf31877f91005ded8825486d906fca584f9ecb"
Jul 17 22:15:30.331356 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:30.331176    2565 scope.go:117] "RemoveContainer" containerID="7ae3e1b2f0052a82da46a298bd0a5188796d6579ac8b7d3d0bbd11f51358d760"
Jul 17 22:15:30.331484 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:15:30.331403    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87_odh-ai-gateway-infra(e339252c-d34f-4029-9619-68321503eea2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87" podUID="e339252c-d34f-4029-9619-68321503eea2"
Jul 17 22:15:31.336552 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:31.336527    2565 scope.go:117] "RemoveContainer" containerID="7ae3e1b2f0052a82da46a298bd0a5188796d6579ac8b7d3d0bbd11f51358d760"
Jul 17 22:15:31.337019 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:15:31.336750    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87_odh-ai-gateway-infra(e339252c-d34f-4029-9619-68321503eea2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87" podUID="e339252c-d34f-4029-9619-68321503eea2"
Jul 17 22:15:37.213610 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:37.213582    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:15:37.217928 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:37.217901    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:15:43.176601 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:43.176566    2565 scope.go:117] "RemoveContainer" containerID="7ae3e1b2f0052a82da46a298bd0a5188796d6579ac8b7d3d0bbd11f51358d760"
Jul 17 22:15:44.399648 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:44.399612    2565 generic.go:356] "Generic (PLEG): container finished" podID="e339252c-d34f-4029-9619-68321503eea2" containerID="4725c3177a395b20d4a23dbc0a8b782cf9cad5a65a96f2aea7de8ad2bd4c78a2" exitCode=7
Jul 17 22:15:44.400092 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:44.399693    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87" event={"ID":"e339252c-d34f-4029-9619-68321503eea2","Type":"ContainerDied","Data":"4725c3177a395b20d4a23dbc0a8b782cf9cad5a65a96f2aea7de8ad2bd4c78a2"}
Jul 17 22:15:44.400092 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:44.399759    2565 scope.go:117] "RemoveContainer" containerID="7ae3e1b2f0052a82da46a298bd0a5188796d6579ac8b7d3d0bbd11f51358d760"
Jul 17 22:15:44.400223 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:44.400185    2565 scope.go:117] "RemoveContainer" containerID="4725c3177a395b20d4a23dbc0a8b782cf9cad5a65a96f2aea7de8ad2bd4c78a2"
Jul 17 22:15:44.400443 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:15:44.400424    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87_odh-ai-gateway-infra(e339252c-d34f-4029-9619-68321503eea2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87" podUID="e339252c-d34f-4029-9619-68321503eea2"
Jul 17 22:15:45.426415 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:45.426383    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87"]
Jul 17 22:15:45.567044 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:45.567012    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87"
Jul 17 22:15:45.613581 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:45.613549    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e339252c-d34f-4029-9619-68321503eea2-maas-api-service-ca\") pod \"e339252c-d34f-4029-9619-68321503eea2\" (UID: \"e339252c-d34f-4029-9619-68321503eea2\") "
Jul 17 22:15:45.613900 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:45.613880    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e339252c-d34f-4029-9619-68321503eea2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e339252c-d34f-4029-9619-68321503eea2" (UID: "e339252c-d34f-4029-9619-68321503eea2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:15:45.714680 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:45.714595    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e339252c-d34f-4029-9619-68321503eea2-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:15:46.412356 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:46.412318    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87" event={"ID":"e339252c-d34f-4029-9619-68321503eea2","Type":"ContainerDied","Data":"be511329b43a1269ec015ed77dc9c0307952d008b5e2264b571dd8e6668660f5"}
Jul 17 22:15:46.412356 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:46.412337    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87"
Jul 17 22:15:46.412605 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:46.412391    2565 scope.go:117] "RemoveContainer" containerID="4725c3177a395b20d4a23dbc0a8b782cf9cad5a65a96f2aea7de8ad2bd4c78a2"
Jul 17 22:15:46.433942 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:46.433908    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87"]
Jul 17 22:15:46.439304 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:46.439273    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-n4r87"]
Jul 17 22:15:47.055764 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:47.055731    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:15:47.155220 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:47.155190    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:15:47.183839 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:47.183804    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e339252c-d34f-4029-9619-68321503eea2" path="/var/lib/kubelet/pods/e339252c-d34f-4029-9619-68321503eea2/volumes"
Jul 17 22:15:47.350321 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:47.350218    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"]
Jul 17 22:15:47.350499 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:47.350478    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz" podUID="3b60617e-7f8f-4a38-9bce-07277bee1db2" containerName="istio-proxy" containerID="cri-o://33852645e09692a0019cc768230397fd3f381fa6ca731ef64c5d3e172f284732" gracePeriod=30
Jul 17 22:15:47.379186 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:47.379147    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:15:47.479476 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:47.479440    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:15:48.376747 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:48.376707    2565 patch_prober.go:28] interesting pod/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 22:15:48.376943 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:48.376785    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz" podUID="3b60617e-7f8f-4a38-9bce-07277bee1db2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 22:15:50.047525 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:50.047488    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:15:52.615901 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.615835    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:15:52.674875 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.674841    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-podinfo\") pod \"3b60617e-7f8f-4a38-9bce-07277bee1db2\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") "
Jul 17 22:15:52.675034 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.674901    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88jqp\" (UniqueName: \"kubernetes.io/projected/3b60617e-7f8f-4a38-9bce-07277bee1db2-kube-api-access-88jqp\") pod \"3b60617e-7f8f-4a38-9bce-07277bee1db2\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") "
Jul 17 22:15:52.675034 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.674935    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-credential-socket\") pod \"3b60617e-7f8f-4a38-9bce-07277bee1db2\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") "
Jul 17 22:15:52.675034 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.674970    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-envoy\") pod \"3b60617e-7f8f-4a38-9bce-07277bee1db2\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") "
Jul 17 22:15:52.675034 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.675020    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b60617e-7f8f-4a38-9bce-07277bee1db2-istiod-ca-cert\") pod \"3b60617e-7f8f-4a38-9bce-07277bee1db2\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") "
Jul 17 22:15:52.675239 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.675049    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-data\") pod \"3b60617e-7f8f-4a38-9bce-07277bee1db2\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") "
Jul 17 22:15:52.675239 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.675124    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-workload-socket\") pod \"3b60617e-7f8f-4a38-9bce-07277bee1db2\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") "
Jul 17 22:15:52.675239 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.675152    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-workload-certs\") pod \"3b60617e-7f8f-4a38-9bce-07277bee1db2\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") "
Jul 17 22:15:52.675239 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.675188    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-token\") pod \"3b60617e-7f8f-4a38-9bce-07277bee1db2\" (UID: \"3b60617e-7f8f-4a38-9bce-07277bee1db2\") "
Jul 17 22:15:52.675239 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.675218    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3b60617e-7f8f-4a38-9bce-07277bee1db2" (UID: "3b60617e-7f8f-4a38-9bce-07277bee1db2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:15:52.675507 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.675403    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3b60617e-7f8f-4a38-9bce-07277bee1db2" (UID: "3b60617e-7f8f-4a38-9bce-07277bee1db2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:15:52.675566 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.675510    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-credential-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:15:52.675566 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.675529    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-workload-certs\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:15:52.675566 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.675529    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b60617e-7f8f-4a38-9bce-07277bee1db2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3b60617e-7f8f-4a38-9bce-07277bee1db2" (UID: "3b60617e-7f8f-4a38-9bce-07277bee1db2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:15:52.676094 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.676043    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3b60617e-7f8f-4a38-9bce-07277bee1db2" (UID: "3b60617e-7f8f-4a38-9bce-07277bee1db2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:15:52.676440 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.676411    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-data" (OuterVolumeSpecName: "istio-data") pod "3b60617e-7f8f-4a38-9bce-07277bee1db2" (UID: "3b60617e-7f8f-4a38-9bce-07277bee1db2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:15:52.677386 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.677355    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3b60617e-7f8f-4a38-9bce-07277bee1db2" (UID: "3b60617e-7f8f-4a38-9bce-07277bee1db2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:15:52.677493 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.677454    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3b60617e-7f8f-4a38-9bce-07277bee1db2" (UID: "3b60617e-7f8f-4a38-9bce-07277bee1db2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:15:52.677643 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.677621    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b60617e-7f8f-4a38-9bce-07277bee1db2-kube-api-access-88jqp" (OuterVolumeSpecName: "kube-api-access-88jqp") pod "3b60617e-7f8f-4a38-9bce-07277bee1db2" (UID: "3b60617e-7f8f-4a38-9bce-07277bee1db2"). InnerVolumeSpecName "kube-api-access-88jqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:15:52.677838 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.677819    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-token" (OuterVolumeSpecName: "istio-token") pod "3b60617e-7f8f-4a38-9bce-07277bee1db2" (UID: "3b60617e-7f8f-4a38-9bce-07277bee1db2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:15:52.776520 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.776481    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-token\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:15:52.776520 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.776515    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-podinfo\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:15:52.776520 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.776525    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88jqp\" (UniqueName: \"kubernetes.io/projected/3b60617e-7f8f-4a38-9bce-07277bee1db2-kube-api-access-88jqp\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:15:52.776811 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.776533    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-envoy\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:15:52.776811 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.776542    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b60617e-7f8f-4a38-9bce-07277bee1db2-istiod-ca-cert\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:15:52.776811 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.776551    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-istio-data\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:15:52.776811 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:52.776558    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b60617e-7f8f-4a38-9bce-07277bee1db2-workload-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:15:53.051294 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:53.051264    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:15:53.448560 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:53.448455    2565 generic.go:356] "Generic (PLEG): container finished" podID="3b60617e-7f8f-4a38-9bce-07277bee1db2" containerID="33852645e09692a0019cc768230397fd3f381fa6ca731ef64c5d3e172f284732" exitCode=0
Jul 17 22:15:53.448560 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:53.448496    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"
Jul 17 22:15:53.448759 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:53.448493    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz" event={"ID":"3b60617e-7f8f-4a38-9bce-07277bee1db2","Type":"ContainerDied","Data":"33852645e09692a0019cc768230397fd3f381fa6ca731ef64c5d3e172f284732"}
Jul 17 22:15:53.448759 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:53.448626    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz" event={"ID":"3b60617e-7f8f-4a38-9bce-07277bee1db2","Type":"ContainerDied","Data":"db4439daf25e3989353bb08204d49707f1f6eddef70d0298bf2989407e7aa781"}
Jul 17 22:15:53.448759 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:53.448663    2565 scope.go:117] "RemoveContainer" containerID="33852645e09692a0019cc768230397fd3f381fa6ca731ef64c5d3e172f284732"
Jul 17 22:15:53.457687 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:53.457667    2565 scope.go:117] "RemoveContainer" containerID="db866da2aa79424d22543d81c7d7537ddb67422afe167c9195622ad832ea5392"
Jul 17 22:15:53.470021 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:53.469766    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"]
Jul 17 22:15:53.470021 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:53.469841    2565 scope.go:117] "RemoveContainer" containerID="33852645e09692a0019cc768230397fd3f381fa6ca731ef64c5d3e172f284732"
Jul 17 22:15:53.470259 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:15:53.470239    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33852645e09692a0019cc768230397fd3f381fa6ca731ef64c5d3e172f284732\": container with ID starting with 33852645e09692a0019cc768230397fd3f381fa6ca731ef64c5d3e172f284732 not found: ID does not exist" containerID="33852645e09692a0019cc768230397fd3f381fa6ca731ef64c5d3e172f284732"
Jul 17 22:15:53.470308 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:53.470269    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33852645e09692a0019cc768230397fd3f381fa6ca731ef64c5d3e172f284732"} err="failed to get container status \"33852645e09692a0019cc768230397fd3f381fa6ca731ef64c5d3e172f284732\": rpc error: code = NotFound desc = could not find container \"33852645e09692a0019cc768230397fd3f381fa6ca731ef64c5d3e172f284732\": container with ID starting with 33852645e09692a0019cc768230397fd3f381fa6ca731ef64c5d3e172f284732 not found: ID does not exist"
Jul 17 22:15:53.470308 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:53.470288    2565 scope.go:117] "RemoveContainer" containerID="db866da2aa79424d22543d81c7d7537ddb67422afe167c9195622ad832ea5392"
Jul 17 22:15:53.470579 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:15:53.470562    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db866da2aa79424d22543d81c7d7537ddb67422afe167c9195622ad832ea5392\": container with ID starting with db866da2aa79424d22543d81c7d7537ddb67422afe167c9195622ad832ea5392 not found: ID does not exist" containerID="db866da2aa79424d22543d81c7d7537ddb67422afe167c9195622ad832ea5392"
Jul 17 22:15:53.470639 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:53.470589    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db866da2aa79424d22543d81c7d7537ddb67422afe167c9195622ad832ea5392"} err="failed to get container status \"db866da2aa79424d22543d81c7d7537ddb67422afe167c9195622ad832ea5392\": rpc error: code = NotFound desc = could not find container \"db866da2aa79424d22543d81c7d7537ddb67422afe167c9195622ad832ea5392\": container with ID starting with db866da2aa79424d22543d81c7d7537ddb67422afe167c9195622ad832ea5392 not found: ID does not exist"
Jul 17 22:15:53.474404 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:53.474379    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7fb802f2-openshift-default-6984dbdb85-s5tjz"]
Jul 17 22:15:53.869319 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:53.869285    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:15:55.182894 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:55.182856    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b60617e-7f8f-4a38-9bce-07277bee1db2" path="/var/lib/kubelet/pods/3b60617e-7f8f-4a38-9bce-07277bee1db2/volumes"
Jul 17 22:15:58.364452 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:58.364418    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:15:58.464335 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:58.464299    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:15:59.374645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.374593    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:15:59.439152 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.439114    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:15:59.678215 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.678124    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:15:59.778125 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.778059    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:15:59.825317 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.825284    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"]
Jul 17 22:15:59.825640 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.825627    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e339252c-d34f-4029-9619-68321503eea2" containerName="revoke-keys"
Jul 17 22:15:59.825640 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.825640    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e339252c-d34f-4029-9619-68321503eea2" containerName="revoke-keys"
Jul 17 22:15:59.825769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.825651    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b60617e-7f8f-4a38-9bce-07277bee1db2" containerName="istio-proxy"
Jul 17 22:15:59.825769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.825656    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b60617e-7f8f-4a38-9bce-07277bee1db2" containerName="istio-proxy"
Jul 17 22:15:59.825769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.825671    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e339252c-d34f-4029-9619-68321503eea2" containerName="revoke-keys"
Jul 17 22:15:59.825769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.825676    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e339252c-d34f-4029-9619-68321503eea2" containerName="revoke-keys"
Jul 17 22:15:59.825769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.825683    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e339252c-d34f-4029-9619-68321503eea2" containerName="revoke-keys"
Jul 17 22:15:59.825769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.825689    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e339252c-d34f-4029-9619-68321503eea2" containerName="revoke-keys"
Jul 17 22:15:59.825769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.825740    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e339252c-d34f-4029-9619-68321503eea2" containerName="revoke-keys"
Jul 17 22:15:59.825769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.825748    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b60617e-7f8f-4a38-9bce-07277bee1db2" containerName="istio-proxy"
Jul 17 22:15:59.825769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.825756    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e339252c-d34f-4029-9619-68321503eea2" containerName="revoke-keys"
Jul 17 22:15:59.830352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.830327    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:15:59.832641 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.832617    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-09e18326-gw-openshift-default-dockercfg-2vfrg\""
Jul 17 22:15:59.846323 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.846289    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"]
Jul 17 22:15:59.862701 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.862656    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:15:59.946794 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.946697    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-credential-socket\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:15:59.946794 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.946751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-podinfo\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:15:59.947016 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.946824    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jskwz\" (UniqueName: \"kubernetes.io/projected/249e93ae-028e-41d5-b989-58f4289e7d9e-kube-api-access-jskwz\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:15:59.947016 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.946909    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-workload-socket\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:15:59.947016 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.946941    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-data\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:15:59.947016 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.946964    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249e93ae-028e-41d5-b989-58f4289e7d9e-istiod-ca-cert\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:15:59.947016 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.946999    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-envoy\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:15:59.947264 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.947022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-token\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:15:59.947264 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:15:59.947109    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-workload-certs\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.048175 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.048132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jskwz\" (UniqueName: \"kubernetes.io/projected/249e93ae-028e-41d5-b989-58f4289e7d9e-kube-api-access-jskwz\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.048383 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.048203    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-workload-socket\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.048383 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.048231    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-data\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.048383 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.048252    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249e93ae-028e-41d5-b989-58f4289e7d9e-istiod-ca-cert\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.048383 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.048278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-envoy\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.048577 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.048423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-token\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.048577 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.048529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-workload-certs\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.048678 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.048583    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-credential-socket\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.048678 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.048612    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-podinfo\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.048678 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.048646    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-data\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.048831 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.048680    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-workload-socket\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.048930 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.048910    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-workload-certs\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.049171 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.048945    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-credential-socket\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.049171 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.049019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249e93ae-028e-41d5-b989-58f4289e7d9e-istiod-ca-cert\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.050720 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.050690    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-envoy\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.051028 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.051010    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-podinfo\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.056643 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.056615    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jskwz\" (UniqueName: \"kubernetes.io/projected/249e93ae-028e-41d5-b989-58f4289e7d9e-kube-api-access-jskwz\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.057054 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.057034    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-token\") pod \"e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") " pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.148172 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.148128    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:00.291671 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.291643    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"]
Jul 17 22:16:00.294092 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:16:00.294039    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249e93ae_028e_41d5_b989_58f4289e7d9e.slice/crio-039ded588ac4a8372162d8d738149b534f51068bac85a90fb7c37ddb0257704f WatchSource:0}: Error finding container 039ded588ac4a8372162d8d738149b534f51068bac85a90fb7c37ddb0257704f: Status 404 returned error can't find the container with id 039ded588ac4a8372162d8d738149b534f51068bac85a90fb7c37ddb0257704f
Jul 17 22:16:00.296305 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.296266    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:16:00.296378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.296361    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:16:00.296422 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.296401    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:16:00.485043 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.484952    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v" event={"ID":"249e93ae-028e-41d5-b989-58f4289e7d9e","Type":"ContainerStarted","Data":"faa527cfd251e4379e4ba737996681e8177c525a0775219fdf575b43931ee3ee"}
Jul 17 22:16:00.485043 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.485014    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v" event={"ID":"249e93ae-028e-41d5-b989-58f4289e7d9e","Type":"ContainerStarted","Data":"039ded588ac4a8372162d8d738149b534f51068bac85a90fb7c37ddb0257704f"}
Jul 17 22:16:00.507571 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:00.507518    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v" podStartSLOduration=1.507500813 podStartE2EDuration="1.507500813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:16:00.50603003 +0000 UTC m=+1823.895046683" watchObservedRunningTime="2026-07-17 22:16:00.507500813 +0000 UTC m=+1823.896517463"
Jul 17 22:16:01.149125 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:01.149085    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:01.153896 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:01.153870    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:01.181749 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:01.181717    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:16:01.293441 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:01.293400    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:16:01.394084 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:01.394030    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:16:01.490032 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:01.489939    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:01.490943 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:01.490926    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:01.948587 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:01.948556    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:16:02.049619 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:02.049584    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:16:03.929938 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:03.929904    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:16:04.000613 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:04.000581    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:16:05.593049 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:05.593000    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x"]
Jul 17 22:16:05.593591 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:05.593567    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b60617e-7f8f-4a38-9bce-07277bee1db2" containerName="istio-proxy"
Jul 17 22:16:05.593672 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:05.593595    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b60617e-7f8f-4a38-9bce-07277bee1db2" containerName="istio-proxy"
Jul 17 22:16:05.593725 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:05.593674    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e339252c-d34f-4029-9619-68321503eea2" containerName="revoke-keys"
Jul 17 22:16:05.593725 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:05.593686    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b60617e-7f8f-4a38-9bce-07277bee1db2" containerName="istio-proxy"
Jul 17 22:16:05.596992 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:05.596970    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x"
Jul 17 22:16:05.599572 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:05.599544    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:16:05.599709 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:05.599593    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:16:05.604240 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:05.604216    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x"]
Jul 17 22:16:05.702171 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:05.702121    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10aec55-4e36-4a1a-abd9-94187f4aefb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x\" (UID: \"d10aec55-4e36-4a1a-abd9-94187f4aefb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x"
Jul 17 22:16:05.803607 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:05.803568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10aec55-4e36-4a1a-abd9-94187f4aefb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x\" (UID: \"d10aec55-4e36-4a1a-abd9-94187f4aefb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x"
Jul 17 22:16:05.804269 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:05.804244    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10aec55-4e36-4a1a-abd9-94187f4aefb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x\" (UID: \"d10aec55-4e36-4a1a-abd9-94187f4aefb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x"
Jul 17 22:16:05.909480 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:05.909382    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x"
Jul 17 22:16:06.055623 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:06.055599    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x"]
Jul 17 22:16:06.058253 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:16:06.058213    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10aec55_4e36_4a1a_abd9_94187f4aefb0.slice/crio-5eada60da10dc27ad8240cf0d39fe6345e89b7ced672fe6bfee66184a6c506ee WatchSource:0}: Error finding container 5eada60da10dc27ad8240cf0d39fe6345e89b7ced672fe6bfee66184a6c506ee: Status 404 returned error can't find the container with id 5eada60da10dc27ad8240cf0d39fe6345e89b7ced672fe6bfee66184a6c506ee
Jul 17 22:16:06.517045 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:06.517013    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x" event={"ID":"d10aec55-4e36-4a1a-abd9-94187f4aefb0","Type":"ContainerStarted","Data":"c8c926c6a5b4290001308ce94c919d0373619937d0ad368a01ba0f538d8521e9"}
Jul 17 22:16:06.517257 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:06.517088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x" event={"ID":"d10aec55-4e36-4a1a-abd9-94187f4aefb0","Type":"ContainerStarted","Data":"5eada60da10dc27ad8240cf0d39fe6345e89b7ced672fe6bfee66184a6c506ee"}
Jul 17 22:16:06.542908 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:06.542794    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x" podStartSLOduration=1.542777609 podStartE2EDuration="1.542777609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:16:06.540199935 +0000 UTC m=+1829.929216585" watchObservedRunningTime="2026-07-17 22:16:06.542777609 +0000 UTC m=+1829.931794260"
Jul 17 22:16:07.491599 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:07.491558    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:16:07.523243 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:07.523208    2565 generic.go:356] "Generic (PLEG): container finished" podID="d10aec55-4e36-4a1a-abd9-94187f4aefb0" containerID="c8c926c6a5b4290001308ce94c919d0373619937d0ad368a01ba0f538d8521e9" exitCode=7
Jul 17 22:16:07.523395 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:07.523252    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x" event={"ID":"d10aec55-4e36-4a1a-abd9-94187f4aefb0","Type":"ContainerDied","Data":"c8c926c6a5b4290001308ce94c919d0373619937d0ad368a01ba0f538d8521e9"}
Jul 17 22:16:07.523495 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:07.523483    2565 scope.go:117] "RemoveContainer" containerID="c8c926c6a5b4290001308ce94c919d0373619937d0ad368a01ba0f538d8521e9"
Jul 17 22:16:08.529599 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:08.529563    2565 generic.go:356] "Generic (PLEG): container finished" podID="d10aec55-4e36-4a1a-abd9-94187f4aefb0" containerID="9c604b1c9833194addc927afeb90eca1b0a46aeacb7bf0d3e4a01c5442003a52" exitCode=7
Jul 17 22:16:08.530054 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:08.529616    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x" event={"ID":"d10aec55-4e36-4a1a-abd9-94187f4aefb0","Type":"ContainerDied","Data":"9c604b1c9833194addc927afeb90eca1b0a46aeacb7bf0d3e4a01c5442003a52"}
Jul 17 22:16:08.530054 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:08.529665    2565 scope.go:117] "RemoveContainer" containerID="c8c926c6a5b4290001308ce94c919d0373619937d0ad368a01ba0f538d8521e9"
Jul 17 22:16:08.530054 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:08.529840    2565 scope.go:117] "RemoveContainer" containerID="9c604b1c9833194addc927afeb90eca1b0a46aeacb7bf0d3e4a01c5442003a52"
Jul 17 22:16:08.530229 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:16:08.530119    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x_odh-ai-gateway-infra(d10aec55-4e36-4a1a-abd9-94187f4aefb0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x" podUID="d10aec55-4e36-4a1a-abd9-94187f4aefb0"
Jul 17 22:16:09.535499 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:09.535459    2565 scope.go:117] "RemoveContainer" containerID="9c604b1c9833194addc927afeb90eca1b0a46aeacb7bf0d3e4a01c5442003a52"
Jul 17 22:16:09.535902 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:16:09.535670    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x_odh-ai-gateway-infra(d10aec55-4e36-4a1a-abd9-94187f4aefb0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x" podUID="d10aec55-4e36-4a1a-abd9-94187f4aefb0"
Jul 17 22:16:09.571904 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:09.571871    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:16:17.360489 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:17.359601    2565 patch_prober.go:28] interesting pod/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": read tcp 10.133.0.2:60596->10.133.0.40:15021: read: connection reset by peer" start-of-body=
Jul 17 22:16:17.360489 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:17.359673    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v" podUID="249e93ae-028e-41d5-b989-58f4289e7d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": read tcp 10.133.0.2:60596->10.133.0.40:15021: read: connection reset by peer"
Jul 17 22:16:17.435426 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:17.435381    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd"]
Jul 17 22:16:17.438869 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:17.438842    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd"
Jul 17 22:16:17.456348 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:17.456309    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd"]
Jul 17 22:16:17.524019 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:17.523924    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7775d17-ba7d-4d63-8e51-4e5995a6098d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd\" (UID: \"d7775d17-ba7d-4d63-8e51-4e5995a6098d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd"
Jul 17 22:16:17.575256 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:17.575220    2565 generic.go:356] "Generic (PLEG): container finished" podID="249e93ae-028e-41d5-b989-58f4289e7d9e" containerID="faa527cfd251e4379e4ba737996681e8177c525a0775219fdf575b43931ee3ee" exitCode=0
Jul 17 22:16:17.575436 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:17.575300    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v" event={"ID":"249e93ae-028e-41d5-b989-58f4289e7d9e","Type":"ContainerDied","Data":"faa527cfd251e4379e4ba737996681e8177c525a0775219fdf575b43931ee3ee"}
Jul 17 22:16:17.575798 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:17.575751    2565 scope.go:117] "RemoveContainer" containerID="faa527cfd251e4379e4ba737996681e8177c525a0775219fdf575b43931ee3ee"
Jul 17 22:16:17.576898 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:17.576869    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:16:17.577041 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:17.577014    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:16:17.577249 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:17.577223    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:16:17.624832 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:17.624798    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7775d17-ba7d-4d63-8e51-4e5995a6098d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd\" (UID: \"d7775d17-ba7d-4d63-8e51-4e5995a6098d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd"
Jul 17 22:16:17.625583 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:17.625558    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7775d17-ba7d-4d63-8e51-4e5995a6098d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd\" (UID: \"d7775d17-ba7d-4d63-8e51-4e5995a6098d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd"
Jul 17 22:16:17.748720 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:17.748685    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd"
Jul 17 22:16:17.906503 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:17.906414    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd"]
Jul 17 22:16:17.910637 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:16:17.910603    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7775d17_ba7d_4d63_8e51_4e5995a6098d.slice/crio-ba95995f22b4747148f8b88bd2bf30ab5aafd57697dd33e8fa7359b8ac9d19ec WatchSource:0}: Error finding container ba95995f22b4747148f8b88bd2bf30ab5aafd57697dd33e8fa7359b8ac9d19ec: Status 404 returned error can't find the container with id ba95995f22b4747148f8b88bd2bf30ab5aafd57697dd33e8fa7359b8ac9d19ec
Jul 17 22:16:18.586450 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:18.586407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd" event={"ID":"d7775d17-ba7d-4d63-8e51-4e5995a6098d","Type":"ContainerStarted","Data":"ab67efb890a97901278d7384341de69488ab88e495fc1982f4b8e52b07ef647b"}
Jul 17 22:16:18.586874 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:18.586477    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd" event={"ID":"d7775d17-ba7d-4d63-8e51-4e5995a6098d","Type":"ContainerStarted","Data":"ba95995f22b4747148f8b88bd2bf30ab5aafd57697dd33e8fa7359b8ac9d19ec"}
Jul 17 22:16:18.588688 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:18.588664    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v" event={"ID":"249e93ae-028e-41d5-b989-58f4289e7d9e","Type":"ContainerStarted","Data":"c797cdae8427d2454fc5fc46b2242db10a0245a58a38d30ad35830a81eac080f"}
Jul 17 22:16:18.629595 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:18.629542    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd" podStartSLOduration=1.6295257730000001 podStartE2EDuration="1.629525773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:16:18.626293589 +0000 UTC m=+1842.015310240" watchObservedRunningTime="2026-07-17 22:16:18.629525773 +0000 UTC m=+1842.018542464"
Jul 17 22:16:19.148513 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:19.148475    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:19.149033 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:19.149006    2565 patch_prober.go:28] interesting pod/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Jul 17 22:16:19.149233 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:19.149059    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v" podUID="249e93ae-028e-41d5-b989-58f4289e7d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Jul 17 22:16:19.594803 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:19.594764    2565 generic.go:356] "Generic (PLEG): container finished" podID="d7775d17-ba7d-4d63-8e51-4e5995a6098d" containerID="ab67efb890a97901278d7384341de69488ab88e495fc1982f4b8e52b07ef647b" exitCode=6
Jul 17 22:16:19.595238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:19.594843    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd" event={"ID":"d7775d17-ba7d-4d63-8e51-4e5995a6098d","Type":"ContainerDied","Data":"ab67efb890a97901278d7384341de69488ab88e495fc1982f4b8e52b07ef647b"}
Jul 17 22:16:19.595238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:19.595044    2565 scope.go:117] "RemoveContainer" containerID="ab67efb890a97901278d7384341de69488ab88e495fc1982f4b8e52b07ef647b"
Jul 17 22:16:20.148756 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:20.148661    2565 patch_prober.go:28] interesting pod/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Jul 17 22:16:20.148756 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:20.148727    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v" podUID="249e93ae-028e-41d5-b989-58f4289e7d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Jul 17 22:16:20.601254 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:20.601218    2565 generic.go:356] "Generic (PLEG): container finished" podID="d7775d17-ba7d-4d63-8e51-4e5995a6098d" containerID="c5eba5d42e29efa2e3d471c51c010e6d2ee982e79819a8a0e2a45c0d16ce6bf1" exitCode=6
Jul 17 22:16:20.601680 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:20.601302    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd" event={"ID":"d7775d17-ba7d-4d63-8e51-4e5995a6098d","Type":"ContainerDied","Data":"c5eba5d42e29efa2e3d471c51c010e6d2ee982e79819a8a0e2a45c0d16ce6bf1"}
Jul 17 22:16:20.601680 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:20.601371    2565 scope.go:117] "RemoveContainer" containerID="ab67efb890a97901278d7384341de69488ab88e495fc1982f4b8e52b07ef647b"
Jul 17 22:16:20.601680 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:20.601611    2565 scope.go:117] "RemoveContainer" containerID="c5eba5d42e29efa2e3d471c51c010e6d2ee982e79819a8a0e2a45c0d16ce6bf1"
Jul 17 22:16:20.601828 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:16:20.601810    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd_odh-ai-gateway-infra(d7775d17-ba7d-4d63-8e51-4e5995a6098d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd" podUID="d7775d17-ba7d-4d63-8e51-4e5995a6098d"
Jul 17 22:16:21.149333 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:21.149289    2565 patch_prober.go:28] interesting pod/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Jul 17 22:16:21.149518 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:21.149348    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v" podUID="249e93ae-028e-41d5-b989-58f4289e7d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Jul 17 22:16:21.608160 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:21.608131    2565 scope.go:117] "RemoveContainer" containerID="c5eba5d42e29efa2e3d471c51c010e6d2ee982e79819a8a0e2a45c0d16ce6bf1"
Jul 17 22:16:21.608653 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:16:21.608399    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd_odh-ai-gateway-infra(d7775d17-ba7d-4d63-8e51-4e5995a6098d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd" podUID="d7775d17-ba7d-4d63-8e51-4e5995a6098d"
Jul 17 22:16:22.148545 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:22.148505    2565 patch_prober.go:28] interesting pod/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Jul 17 22:16:22.148734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:22.148564    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v" podUID="249e93ae-028e-41d5-b989-58f4289e7d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Jul 17 22:16:23.153545 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:23.153517    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:23.153976 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:23.153846    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:23.154601 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:23.154582    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:16:24.176587 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:24.176550    2565 scope.go:117] "RemoveContainer" containerID="9c604b1c9833194addc927afeb90eca1b0a46aeacb7bf0d3e4a01c5442003a52"
Jul 17 22:16:24.622844 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:24.622810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x" event={"ID":"d10aec55-4e36-4a1a-abd9-94187f4aefb0","Type":"ContainerStarted","Data":"6639c744005fa10c20fc30002b5c51f64ec15af8e9f8aff769357d6aab6c4fa8"}
Jul 17 22:16:25.630291 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:25.630254    2565 generic.go:356] "Generic (PLEG): container finished" podID="d10aec55-4e36-4a1a-abd9-94187f4aefb0" containerID="6639c744005fa10c20fc30002b5c51f64ec15af8e9f8aff769357d6aab6c4fa8" exitCode=7
Jul 17 22:16:25.630786 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:25.630330    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x" event={"ID":"d10aec55-4e36-4a1a-abd9-94187f4aefb0","Type":"ContainerDied","Data":"6639c744005fa10c20fc30002b5c51f64ec15af8e9f8aff769357d6aab6c4fa8"}
Jul 17 22:16:25.630786 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:25.630411    2565 scope.go:117] "RemoveContainer" containerID="9c604b1c9833194addc927afeb90eca1b0a46aeacb7bf0d3e4a01c5442003a52"
Jul 17 22:16:25.630786 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:25.630745    2565 scope.go:117] "RemoveContainer" containerID="6639c744005fa10c20fc30002b5c51f64ec15af8e9f8aff769357d6aab6c4fa8"
Jul 17 22:16:25.630986 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:16:25.630962    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x_odh-ai-gateway-infra(d10aec55-4e36-4a1a-abd9-94187f4aefb0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x" podUID="d10aec55-4e36-4a1a-abd9-94187f4aefb0"
Jul 17 22:16:25.665753 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:25.665712    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x"]
Jul 17 22:16:26.785351 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:26.785324    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x"
Jul 17 22:16:26.914105 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:26.913969    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10aec55-4e36-4a1a-abd9-94187f4aefb0-maas-api-service-ca\") pod \"d10aec55-4e36-4a1a-abd9-94187f4aefb0\" (UID: \"d10aec55-4e36-4a1a-abd9-94187f4aefb0\") "
Jul 17 22:16:26.914442 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:26.914414    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10aec55-4e36-4a1a-abd9-94187f4aefb0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d10aec55-4e36-4a1a-abd9-94187f4aefb0" (UID: "d10aec55-4e36-4a1a-abd9-94187f4aefb0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:16:27.014954 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:27.014908    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10aec55-4e36-4a1a-abd9-94187f4aefb0-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:16:27.643554 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:27.643520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x" event={"ID":"d10aec55-4e36-4a1a-abd9-94187f4aefb0","Type":"ContainerDied","Data":"5eada60da10dc27ad8240cf0d39fe6345e89b7ced672fe6bfee66184a6c506ee"}
Jul 17 22:16:27.643732 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:27.643577    2565 scope.go:117] "RemoveContainer" containerID="6639c744005fa10c20fc30002b5c51f64ec15af8e9f8aff769357d6aab6c4fa8"
Jul 17 22:16:27.643732 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:27.643533    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x"
Jul 17 22:16:27.667249 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:27.667212    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x"]
Jul 17 22:16:27.678084 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:27.678027    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-vjg2x"]
Jul 17 22:16:29.182669 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:29.182632    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d10aec55-4e36-4a1a-abd9-94187f4aefb0" path="/var/lib/kubelet/pods/d10aec55-4e36-4a1a-abd9-94187f4aefb0/volumes"
Jul 17 22:16:35.177033 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:35.176998    2565 scope.go:117] "RemoveContainer" containerID="c5eba5d42e29efa2e3d471c51c010e6d2ee982e79819a8a0e2a45c0d16ce6bf1"
Jul 17 22:16:35.684255 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:35.684216    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd" event={"ID":"d7775d17-ba7d-4d63-8e51-4e5995a6098d","Type":"ContainerStarted","Data":"14a97d5789cf669904b9862f47d374f0938d1a72e0cf41e7d28233af9243cb4b"}
Jul 17 22:16:36.690946 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:36.690915    2565 generic.go:356] "Generic (PLEG): container finished" podID="d7775d17-ba7d-4d63-8e51-4e5995a6098d" containerID="14a97d5789cf669904b9862f47d374f0938d1a72e0cf41e7d28233af9243cb4b" exitCode=6
Jul 17 22:16:36.691448 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:36.690953    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd" event={"ID":"d7775d17-ba7d-4d63-8e51-4e5995a6098d","Type":"ContainerDied","Data":"14a97d5789cf669904b9862f47d374f0938d1a72e0cf41e7d28233af9243cb4b"}
Jul 17 22:16:36.691448 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:36.691001    2565 scope.go:117] "RemoveContainer" containerID="c5eba5d42e29efa2e3d471c51c010e6d2ee982e79819a8a0e2a45c0d16ce6bf1"
Jul 17 22:16:36.691448 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:36.691365    2565 scope.go:117] "RemoveContainer" containerID="14a97d5789cf669904b9862f47d374f0938d1a72e0cf41e7d28233af9243cb4b"
Jul 17 22:16:36.691635 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:16:36.691612    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd_odh-ai-gateway-infra(d7775d17-ba7d-4d63-8e51-4e5995a6098d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd" podUID="d7775d17-ba7d-4d63-8e51-4e5995a6098d"
Jul 17 22:16:36.733944 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:36.733907    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd"]
Jul 17 22:16:37.846167 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:37.846139    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd"
Jul 17 22:16:37.915925 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:37.915888    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7775d17-ba7d-4d63-8e51-4e5995a6098d-maas-api-service-ca\") pod \"d7775d17-ba7d-4d63-8e51-4e5995a6098d\" (UID: \"d7775d17-ba7d-4d63-8e51-4e5995a6098d\") "
Jul 17 22:16:37.916312 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:37.916288    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7775d17-ba7d-4d63-8e51-4e5995a6098d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7775d17-ba7d-4d63-8e51-4e5995a6098d" (UID: "d7775d17-ba7d-4d63-8e51-4e5995a6098d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:16:38.016714 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:38.016673    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7775d17-ba7d-4d63-8e51-4e5995a6098d-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:16:38.703697 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:38.703659    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd" event={"ID":"d7775d17-ba7d-4d63-8e51-4e5995a6098d","Type":"ContainerDied","Data":"ba95995f22b4747148f8b88bd2bf30ab5aafd57697dd33e8fa7359b8ac9d19ec"}
Jul 17 22:16:38.703907 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:38.703706    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd"
Jul 17 22:16:38.703907 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:38.703722    2565 scope.go:117] "RemoveContainer" containerID="14a97d5789cf669904b9862f47d374f0938d1a72e0cf41e7d28233af9243cb4b"
Jul 17 22:16:38.727273 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:38.727241    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd"]
Jul 17 22:16:38.730517 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:38.730489    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-csdwd"]
Jul 17 22:16:39.182582 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:16:39.182546    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7775d17-ba7d-4d63-8e51-4e5995a6098d" path="/var/lib/kubelet/pods/d7775d17-ba7d-4d63-8e51-4e5995a6098d/volumes"
Jul 17 22:17:05.645865 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.645781    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5"]
Jul 17 22:17:05.646326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.646163    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d10aec55-4e36-4a1a-abd9-94187f4aefb0" containerName="revoke-keys"
Jul 17 22:17:05.646326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.646176    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d10aec55-4e36-4a1a-abd9-94187f4aefb0" containerName="revoke-keys"
Jul 17 22:17:05.646326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.646186    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d10aec55-4e36-4a1a-abd9-94187f4aefb0" containerName="revoke-keys"
Jul 17 22:17:05.646326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.646192    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d10aec55-4e36-4a1a-abd9-94187f4aefb0" containerName="revoke-keys"
Jul 17 22:17:05.646326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.646208    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7775d17-ba7d-4d63-8e51-4e5995a6098d" containerName="revoke-keys"
Jul 17 22:17:05.646326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.646213    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7775d17-ba7d-4d63-8e51-4e5995a6098d" containerName="revoke-keys"
Jul 17 22:17:05.646326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.646224    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7775d17-ba7d-4d63-8e51-4e5995a6098d" containerName="revoke-keys"
Jul 17 22:17:05.646326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.646229    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7775d17-ba7d-4d63-8e51-4e5995a6098d" containerName="revoke-keys"
Jul 17 22:17:05.646326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.646237    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7775d17-ba7d-4d63-8e51-4e5995a6098d" containerName="revoke-keys"
Jul 17 22:17:05.646326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.646242    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7775d17-ba7d-4d63-8e51-4e5995a6098d" containerName="revoke-keys"
Jul 17 22:17:05.646326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.646298    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d10aec55-4e36-4a1a-abd9-94187f4aefb0" containerName="revoke-keys"
Jul 17 22:17:05.646326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.646306    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7775d17-ba7d-4d63-8e51-4e5995a6098d" containerName="revoke-keys"
Jul 17 22:17:05.646326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.646311    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7775d17-ba7d-4d63-8e51-4e5995a6098d" containerName="revoke-keys"
Jul 17 22:17:05.646326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.646319    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d10aec55-4e36-4a1a-abd9-94187f4aefb0" containerName="revoke-keys"
Jul 17 22:17:05.646326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.646324    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7775d17-ba7d-4d63-8e51-4e5995a6098d" containerName="revoke-keys"
Jul 17 22:17:05.650805 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.650786    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5"
Jul 17 22:17:05.654005 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.653982    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:17:05.654005 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.653994    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:17:05.658643 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.658617    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5"]
Jul 17 22:17:05.704155 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.704119    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"]
Jul 17 22:17:05.704402 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.704358    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v" podUID="249e93ae-028e-41d5-b989-58f4289e7d9e" containerName="istio-proxy" containerID="cri-o://c797cdae8427d2454fc5fc46b2242db10a0245a58a38d30ad35830a81eac080f" gracePeriod=30
Jul 17 22:17:05.733415 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.733352    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:17:05.777567 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.777522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb4f8a6-8112-489a-8b02-a6e2b1f90c53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5\" (UID: \"dfb4f8a6-8112-489a-8b02-a6e2b1f90c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5"
Jul 17 22:17:05.837277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.837236    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:17:05.878459 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.878412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb4f8a6-8112-489a-8b02-a6e2b1f90c53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5\" (UID: \"dfb4f8a6-8112-489a-8b02-a6e2b1f90c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5"
Jul 17 22:17:05.879052 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.879034    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb4f8a6-8112-489a-8b02-a6e2b1f90c53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5\" (UID: \"dfb4f8a6-8112-489a-8b02-a6e2b1f90c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5"
Jul 17 22:17:05.962507 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:05.962416    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5"
Jul 17 22:17:06.109936 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:06.109904    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5"]
Jul 17 22:17:06.112908 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:17:06.112878    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb4f8a6_8112_489a_8b02_a6e2b1f90c53.slice/crio-be0f3e9eb70b0cc97f28606f25c9fa22f023ba473bbc011d83152e4a640c5e43 WatchSource:0}: Error finding container be0f3e9eb70b0cc97f28606f25c9fa22f023ba473bbc011d83152e4a640c5e43: Status 404 returned error can't find the container with id be0f3e9eb70b0cc97f28606f25c9fa22f023ba473bbc011d83152e4a640c5e43
Jul 17 22:17:06.842988 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:06.842949    2565 generic.go:356] "Generic (PLEG): container finished" podID="dfb4f8a6-8112-489a-8b02-a6e2b1f90c53" containerID="670a92478e58417228c4c6262057b5b1f3de349eaea4b2926363b7c228efbcea" exitCode=7
Jul 17 22:17:06.843396 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:06.843039    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5" event={"ID":"dfb4f8a6-8112-489a-8b02-a6e2b1f90c53","Type":"ContainerDied","Data":"670a92478e58417228c4c6262057b5b1f3de349eaea4b2926363b7c228efbcea"}
Jul 17 22:17:06.843396 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:06.843138    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5" event={"ID":"dfb4f8a6-8112-489a-8b02-a6e2b1f90c53","Type":"ContainerStarted","Data":"be0f3e9eb70b0cc97f28606f25c9fa22f023ba473bbc011d83152e4a640c5e43"}
Jul 17 22:17:06.843396 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:06.843223    2565 scope.go:117] "RemoveContainer" containerID="670a92478e58417228c4c6262057b5b1f3de349eaea4b2926363b7c228efbcea"
Jul 17 22:17:07.849980 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:07.849944    2565 generic.go:356] "Generic (PLEG): container finished" podID="dfb4f8a6-8112-489a-8b02-a6e2b1f90c53" containerID="037ae69f381a375534e3c26350ec3589c1885aa97f4cd6271c7f2adf779bf210" exitCode=7
Jul 17 22:17:07.850502 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:07.850025    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5" event={"ID":"dfb4f8a6-8112-489a-8b02-a6e2b1f90c53","Type":"ContainerDied","Data":"037ae69f381a375534e3c26350ec3589c1885aa97f4cd6271c7f2adf779bf210"}
Jul 17 22:17:07.850502 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:07.850127    2565 scope.go:117] "RemoveContainer" containerID="670a92478e58417228c4c6262057b5b1f3de349eaea4b2926363b7c228efbcea"
Jul 17 22:17:07.850502 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:07.850347    2565 scope.go:117] "RemoveContainer" containerID="037ae69f381a375534e3c26350ec3589c1885aa97f4cd6271c7f2adf779bf210"
Jul 17 22:17:07.850670 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:17:07.850606    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5_odh-ai-gateway-infra(dfb4f8a6-8112-489a-8b02-a6e2b1f90c53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5" podUID="dfb4f8a6-8112-489a-8b02-a6e2b1f90c53"
Jul 17 22:17:08.156147 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:08.156033    2565 patch_prober.go:28] interesting pod/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 22:17:08.156147 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:08.156117    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v" podUID="249e93ae-028e-41d5-b989-58f4289e7d9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 22:17:08.646184 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:08.646139    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:17:08.856225 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:08.856200    2565 scope.go:117] "RemoveContainer" containerID="037ae69f381a375534e3c26350ec3589c1885aa97f4cd6271c7f2adf779bf210"
Jul 17 22:17:08.856597 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:17:08.856420    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5_odh-ai-gateway-infra(dfb4f8a6-8112-489a-8b02-a6e2b1f90c53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5" podUID="dfb4f8a6-8112-489a-8b02-a6e2b1f90c53"
Jul 17 22:17:10.867162 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:10.867129    2565 generic.go:356] "Generic (PLEG): container finished" podID="249e93ae-028e-41d5-b989-58f4289e7d9e" containerID="c797cdae8427d2454fc5fc46b2242db10a0245a58a38d30ad35830a81eac080f" exitCode=0
Jul 17 22:17:10.867544 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:10.867205    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v" event={"ID":"249e93ae-028e-41d5-b989-58f4289e7d9e","Type":"ContainerDied","Data":"c797cdae8427d2454fc5fc46b2242db10a0245a58a38d30ad35830a81eac080f"}
Jul 17 22:17:10.867544 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:10.867272    2565 scope.go:117] "RemoveContainer" containerID="faa527cfd251e4379e4ba737996681e8177c525a0775219fdf575b43931ee3ee"
Jul 17 22:17:10.957231 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:10.957206    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:17:11.123530 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.123493    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jskwz\" (UniqueName: \"kubernetes.io/projected/249e93ae-028e-41d5-b989-58f4289e7d9e-kube-api-access-jskwz\") pod \"249e93ae-028e-41d5-b989-58f4289e7d9e\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") "
Jul 17 22:17:11.123739 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.123601    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-token\") pod \"249e93ae-028e-41d5-b989-58f4289e7d9e\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") "
Jul 17 22:17:11.123739 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.123630    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-workload-socket\") pod \"249e93ae-028e-41d5-b989-58f4289e7d9e\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") "
Jul 17 22:17:11.123739 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.123651    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-workload-certs\") pod \"249e93ae-028e-41d5-b989-58f4289e7d9e\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") "
Jul 17 22:17:11.123739 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.123680    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-envoy\") pod \"249e93ae-028e-41d5-b989-58f4289e7d9e\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") "
Jul 17 22:17:11.123739 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.123728    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-podinfo\") pod \"249e93ae-028e-41d5-b989-58f4289e7d9e\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") "
Jul 17 22:17:11.124161 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.123778    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-data\") pod \"249e93ae-028e-41d5-b989-58f4289e7d9e\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") "
Jul 17 22:17:11.124161 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.123808    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-credential-socket\") pod \"249e93ae-028e-41d5-b989-58f4289e7d9e\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") "
Jul 17 22:17:11.124161 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.123835    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249e93ae-028e-41d5-b989-58f4289e7d9e-istiod-ca-cert\") pod \"249e93ae-028e-41d5-b989-58f4289e7d9e\" (UID: \"249e93ae-028e-41d5-b989-58f4289e7d9e\") "
Jul 17 22:17:11.124161 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.123992    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "249e93ae-028e-41d5-b989-58f4289e7d9e" (UID: "249e93ae-028e-41d5-b989-58f4289e7d9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:17:11.124161 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.124006    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "249e93ae-028e-41d5-b989-58f4289e7d9e" (UID: "249e93ae-028e-41d5-b989-58f4289e7d9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:17:11.124583 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.124237    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-workload-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:17:11.124583 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.124258    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-workload-certs\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:17:11.124583 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.124253    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "249e93ae-028e-41d5-b989-58f4289e7d9e" (UID: "249e93ae-028e-41d5-b989-58f4289e7d9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:17:11.124583 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.124321    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/249e93ae-028e-41d5-b989-58f4289e7d9e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "249e93ae-028e-41d5-b989-58f4289e7d9e" (UID: "249e93ae-028e-41d5-b989-58f4289e7d9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:17:11.124792 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.124734    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-data" (OuterVolumeSpecName: "istio-data") pod "249e93ae-028e-41d5-b989-58f4289e7d9e" (UID: "249e93ae-028e-41d5-b989-58f4289e7d9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:17:11.126013 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.125989    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "249e93ae-028e-41d5-b989-58f4289e7d9e" (UID: "249e93ae-028e-41d5-b989-58f4289e7d9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:17:11.126182 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.126122    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249e93ae-028e-41d5-b989-58f4289e7d9e-kube-api-access-jskwz" (OuterVolumeSpecName: "kube-api-access-jskwz") pod "249e93ae-028e-41d5-b989-58f4289e7d9e" (UID: "249e93ae-028e-41d5-b989-58f4289e7d9e"). InnerVolumeSpecName "kube-api-access-jskwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:17:11.126569 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.126555    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "249e93ae-028e-41d5-b989-58f4289e7d9e" (UID: "249e93ae-028e-41d5-b989-58f4289e7d9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:17:11.126627 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.126607    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-token" (OuterVolumeSpecName: "istio-token") pod "249e93ae-028e-41d5-b989-58f4289e7d9e" (UID: "249e93ae-028e-41d5-b989-58f4289e7d9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:17:11.225240 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.225202    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249e93ae-028e-41d5-b989-58f4289e7d9e-istiod-ca-cert\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:17:11.225240 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.225239    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jskwz\" (UniqueName: \"kubernetes.io/projected/249e93ae-028e-41d5-b989-58f4289e7d9e-kube-api-access-jskwz\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:17:11.225240 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.225252    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-token\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:17:11.225496 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.225265    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-envoy\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:17:11.225496 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.225275    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-podinfo\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:17:11.225496 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.225285    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-istio-data\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:17:11.225496 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.225297    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249e93ae-028e-41d5-b989-58f4289e7d9e-credential-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:17:11.585350 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.585311    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:17:11.841792 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.841712    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:17:11.877470 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.877432    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v" event={"ID":"249e93ae-028e-41d5-b989-58f4289e7d9e","Type":"ContainerDied","Data":"039ded588ac4a8372162d8d738149b534f51068bac85a90fb7c37ddb0257704f"}
Jul 17 22:17:11.877470 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.877452    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"
Jul 17 22:17:11.877995 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.877529    2565 scope.go:117] "RemoveContainer" containerID="c797cdae8427d2454fc5fc46b2242db10a0245a58a38d30ad35830a81eac080f"
Jul 17 22:17:11.896875 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.896850    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"]
Jul 17 22:17:11.902722 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:11.902699    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-09e18326-gw-openshift-default-67b5f8968c-sb67v"]
Jul 17 22:17:12.260023 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:12.259987    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:17:13.181770 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:13.181736    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="249e93ae-028e-41d5-b989-58f4289e7d9e" path="/var/lib/kubelet/pods/249e93ae-028e-41d5-b989-58f4289e7d9e/volumes"
Jul 17 22:17:17.472286 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.472250    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5"]
Jul 17 22:17:17.472775 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.472756    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d10aec55-4e36-4a1a-abd9-94187f4aefb0" containerName="revoke-keys"
Jul 17 22:17:17.472866 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.472779    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d10aec55-4e36-4a1a-abd9-94187f4aefb0" containerName="revoke-keys"
Jul 17 22:17:17.472866 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.472798    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="249e93ae-028e-41d5-b989-58f4289e7d9e" containerName="istio-proxy"
Jul 17 22:17:17.472866 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.472807    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="249e93ae-028e-41d5-b989-58f4289e7d9e" containerName="istio-proxy"
Jul 17 22:17:17.472866 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.472816    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="249e93ae-028e-41d5-b989-58f4289e7d9e" containerName="istio-proxy"
Jul 17 22:17:17.472866 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.472823    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="249e93ae-028e-41d5-b989-58f4289e7d9e" containerName="istio-proxy"
Jul 17 22:17:17.473139 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.472939    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="249e93ae-028e-41d5-b989-58f4289e7d9e" containerName="istio-proxy"
Jul 17 22:17:17.473139 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.472956    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d10aec55-4e36-4a1a-abd9-94187f4aefb0" containerName="revoke-keys"
Jul 17 22:17:17.477374 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.477354    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5"
Jul 17 22:17:17.484093 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.484037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a79ff6-639a-4aa4-b320-821f5af64ab9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5\" (UID: \"18a79ff6-639a-4aa4-b320-821f5af64ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5"
Jul 17 22:17:17.484332 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.484192    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5"]
Jul 17 22:17:17.584677 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.584636    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a79ff6-639a-4aa4-b320-821f5af64ab9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5\" (UID: \"18a79ff6-639a-4aa4-b320-821f5af64ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5"
Jul 17 22:17:17.585333 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.585309    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a79ff6-639a-4aa4-b320-821f5af64ab9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5\" (UID: \"18a79ff6-639a-4aa4-b320-821f5af64ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5"
Jul 17 22:17:17.630233 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.630196    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:17:17.789638 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.789598    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5"
Jul 17 22:17:17.843277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.843239    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"]
Jul 17 22:17:17.843882 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.843859    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="249e93ae-028e-41d5-b989-58f4289e7d9e" containerName="istio-proxy"
Jul 17 22:17:17.847539 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.847514    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.851636 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.850772    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-45495ac6-openshift-default-dockercfg-hnjz6\""
Jul 17 22:17:17.877050 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.877013    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"]
Jul 17 22:17:17.886399 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.886363    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-data\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.886399 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.886401    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-credential-socket\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.886636 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.886425    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-envoy\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.886636 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.886452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-workload-certs\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.886636 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.886495    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-podinfo\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.886636 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.886523    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-workload-socket\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.886636 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.886570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ckds\" (UniqueName: \"kubernetes.io/projected/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-kube-api-access-2ckds\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.886636 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.886619    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-token\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.886925 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.886644    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istiod-ca-cert\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.958304 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.958234    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5"]
Jul 17 22:17:17.961036 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:17:17.960999    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a79ff6_639a_4aa4_b320_821f5af64ab9.slice/crio-dbe8143bc15550964e89ec58e6b71a93b3f31e1647fd0dc1b0fa918a75eb4b69 WatchSource:0}: Error finding container dbe8143bc15550964e89ec58e6b71a93b3f31e1647fd0dc1b0fa918a75eb4b69: Status 404 returned error can't find the container with id dbe8143bc15550964e89ec58e6b71a93b3f31e1647fd0dc1b0fa918a75eb4b69
Jul 17 22:17:17.987806 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.987779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-data\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.987910 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.987810    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-credential-socket\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.987910 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.987837    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-envoy\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.987910 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.987860    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-workload-certs\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.987910 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.987898    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-podinfo\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.988105 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.987924    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-workload-socket\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.988105 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.987952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ckds\" (UniqueName: \"kubernetes.io/projected/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-kube-api-access-2ckds\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.988105 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.987981    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-token\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.988105 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.988016    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istiod-ca-cert\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.988320 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.988208    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-credential-socket\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.988413 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.988389    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-data\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.988721 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.988694    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-workload-certs\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.988721 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.988713    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-workload-socket\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.988981 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.988959    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istiod-ca-cert\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.990274 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.990248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-envoy\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:17.990530 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:17.990512    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-podinfo\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:18.002132 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:18.002106    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-token\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:18.002277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:18.002206    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ckds\" (UniqueName: \"kubernetes.io/projected/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-kube-api-access-2ckds\") pod \"e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") " pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:18.181148 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:18.181123    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:18.324827 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:18.324804    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"]
Jul 17 22:17:18.326122 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:17:18.326098    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcfbc0e_024d_41b6_a49d_f759bd3e53b6.slice/crio-44f60e2c7af1a4fa25197030eeaceb89b7ac10af7f09b298da7b34edb6c4ab12 WatchSource:0}: Error finding container 44f60e2c7af1a4fa25197030eeaceb89b7ac10af7f09b298da7b34edb6c4ab12: Status 404 returned error can't find the container with id 44f60e2c7af1a4fa25197030eeaceb89b7ac10af7f09b298da7b34edb6c4ab12
Jul 17 22:17:18.328273 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:18.328237    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:17:18.328360 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:18.328305    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:17:18.328360 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:18.328333    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:17:18.917828 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:18.917789    2565 generic.go:356] "Generic (PLEG): container finished" podID="18a79ff6-639a-4aa4-b320-821f5af64ab9" containerID="3eaa2a44396ea8e16acfa53daec1605ed00e64f6a650109207c4af474e87553b" exitCode=6
Jul 17 22:17:18.918332 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:18.917880    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5" event={"ID":"18a79ff6-639a-4aa4-b320-821f5af64ab9","Type":"ContainerDied","Data":"3eaa2a44396ea8e16acfa53daec1605ed00e64f6a650109207c4af474e87553b"}
Jul 17 22:17:18.918332 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:18.917935    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5" event={"ID":"18a79ff6-639a-4aa4-b320-821f5af64ab9","Type":"ContainerStarted","Data":"dbe8143bc15550964e89ec58e6b71a93b3f31e1647fd0dc1b0fa918a75eb4b69"}
Jul 17 22:17:18.918332 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:18.918147    2565 scope.go:117] "RemoveContainer" containerID="3eaa2a44396ea8e16acfa53daec1605ed00e64f6a650109207c4af474e87553b"
Jul 17 22:17:18.919526 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:18.919505    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts" event={"ID":"edcfbc0e-024d-41b6-a49d-f759bd3e53b6","Type":"ContainerStarted","Data":"30a35eea2ce7ca4bb75d11357bf6392ec8638133d63b3e2aaebf86ebc507247d"}
Jul 17 22:17:18.919617 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:18.919550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts" event={"ID":"edcfbc0e-024d-41b6-a49d-f759bd3e53b6","Type":"ContainerStarted","Data":"44f60e2c7af1a4fa25197030eeaceb89b7ac10af7f09b298da7b34edb6c4ab12"}
Jul 17 22:17:18.954956 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:18.954901    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts" podStartSLOduration=1.9548833860000001 podStartE2EDuration="1.954883386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:17:18.953638159 +0000 UTC m=+1902.342654823" watchObservedRunningTime="2026-07-17 22:17:18.954883386 +0000 UTC m=+1902.343900109"
Jul 17 22:17:19.130477 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:19.130390    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:17:19.184103 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:19.184049    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:19.207792 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:19.207755    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:19.380249 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:19.380212    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:17:19.499358 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:19.499330    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:17:19.926612 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:19.926518    2565 generic.go:356] "Generic (PLEG): container finished" podID="18a79ff6-639a-4aa4-b320-821f5af64ab9" containerID="ef60acfdbe8b5db3638b2919338e8d62ab7b023f8d1bba5c81e491eb6c270661" exitCode=6
Jul 17 22:17:19.927053 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:19.926606    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5" event={"ID":"18a79ff6-639a-4aa4-b320-821f5af64ab9","Type":"ContainerDied","Data":"ef60acfdbe8b5db3638b2919338e8d62ab7b023f8d1bba5c81e491eb6c270661"}
Jul 17 22:17:19.927053 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:19.926663    2565 scope.go:117] "RemoveContainer" containerID="3eaa2a44396ea8e16acfa53daec1605ed00e64f6a650109207c4af474e87553b"
Jul 17 22:17:19.927053 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:19.926821    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:19.927053 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:19.926892    2565 scope.go:117] "RemoveContainer" containerID="ef60acfdbe8b5db3638b2919338e8d62ab7b023f8d1bba5c81e491eb6c270661"
Jul 17 22:17:19.927325 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:17:19.927155    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5_odh-ai-gateway-infra(18a79ff6-639a-4aa4-b320-821f5af64ab9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5" podUID="18a79ff6-639a-4aa4-b320-821f5af64ab9"
Jul 17 22:17:19.928194 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:19.928167    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:19.978551 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:19.978521    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:17:20.072521 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:20.072488    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:17:20.933672 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:20.933646    2565 scope.go:117] "RemoveContainer" containerID="ef60acfdbe8b5db3638b2919338e8d62ab7b023f8d1bba5c81e491eb6c270661"
Jul 17 22:17:20.934096 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:17:20.933817    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5_odh-ai-gateway-infra(18a79ff6-639a-4aa4-b320-821f5af64ab9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5" podUID="18a79ff6-639a-4aa4-b320-821f5af64ab9"
Jul 17 22:17:21.190271 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:21.190186    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:17:21.948438 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:21.948399    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:17:22.177157 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:22.177126    2565 scope.go:117] "RemoveContainer" containerID="037ae69f381a375534e3c26350ec3589c1885aa97f4cd6271c7f2adf779bf210"
Jul 17 22:17:22.945397 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:22.945363    2565 generic.go:356] "Generic (PLEG): container finished" podID="dfb4f8a6-8112-489a-8b02-a6e2b1f90c53" containerID="c4ddef70d1d1eb26f505e8390b622d4d7b68089cb5cb7578ac1eee60d4c99613" exitCode=6
Jul 17 22:17:22.945572 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:22.945413    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5" event={"ID":"dfb4f8a6-8112-489a-8b02-a6e2b1f90c53","Type":"ContainerDied","Data":"c4ddef70d1d1eb26f505e8390b622d4d7b68089cb5cb7578ac1eee60d4c99613"}
Jul 17 22:17:22.945572 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:22.945501    2565 scope.go:117] "RemoveContainer" containerID="037ae69f381a375534e3c26350ec3589c1885aa97f4cd6271c7f2adf779bf210"
Jul 17 22:17:22.945879 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:22.945844    2565 scope.go:117] "RemoveContainer" containerID="c4ddef70d1d1eb26f505e8390b622d4d7b68089cb5cb7578ac1eee60d4c99613"
Jul 17 22:17:22.946130 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:17:22.946107    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5_odh-ai-gateway-infra(dfb4f8a6-8112-489a-8b02-a6e2b1f90c53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5" podUID="dfb4f8a6-8112-489a-8b02-a6e2b1f90c53"
Jul 17 22:17:23.980235 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:23.980186    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5"]
Jul 17 22:17:24.137871 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.137844    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5"
Jul 17 22:17:24.185841 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.185803    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9"]
Jul 17 22:17:24.186631 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.186607    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfb4f8a6-8112-489a-8b02-a6e2b1f90c53" containerName="revoke-keys"
Jul 17 22:17:24.186781 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.186769    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfb4f8a6-8112-489a-8b02-a6e2b1f90c53" containerName="revoke-keys"
Jul 17 22:17:24.186884 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.186874    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfb4f8a6-8112-489a-8b02-a6e2b1f90c53" containerName="revoke-keys"
Jul 17 22:17:24.186962 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.186953    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfb4f8a6-8112-489a-8b02-a6e2b1f90c53" containerName="revoke-keys"
Jul 17 22:17:24.187258 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.187242    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfb4f8a6-8112-489a-8b02-a6e2b1f90c53" containerName="revoke-keys"
Jul 17 22:17:24.187365 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.187354    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfb4f8a6-8112-489a-8b02-a6e2b1f90c53" containerName="revoke-keys"
Jul 17 22:17:24.190471 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.190453    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9"
Jul 17 22:17:24.202931 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.202898    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9"]
Jul 17 22:17:24.245967 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.245938    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb4f8a6-8112-489a-8b02-a6e2b1f90c53-maas-api-service-ca\") pod \"dfb4f8a6-8112-489a-8b02-a6e2b1f90c53\" (UID: \"dfb4f8a6-8112-489a-8b02-a6e2b1f90c53\") "
Jul 17 22:17:24.246213 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.246195    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bec6b6c-93a1-41a5-b875-5152e7c0ce79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9\" (UID: \"8bec6b6c-93a1-41a5-b875-5152e7c0ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9"
Jul 17 22:17:24.246441 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.246414    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb4f8a6-8112-489a-8b02-a6e2b1f90c53-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dfb4f8a6-8112-489a-8b02-a6e2b1f90c53" (UID: "dfb4f8a6-8112-489a-8b02-a6e2b1f90c53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:17:24.347593 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.347553    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bec6b6c-93a1-41a5-b875-5152e7c0ce79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9\" (UID: \"8bec6b6c-93a1-41a5-b875-5152e7c0ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9"
Jul 17 22:17:24.347776 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.347623    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb4f8a6-8112-489a-8b02-a6e2b1f90c53-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:17:24.348235 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.348216    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bec6b6c-93a1-41a5-b875-5152e7c0ce79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9\" (UID: \"8bec6b6c-93a1-41a5-b875-5152e7c0ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9"
Jul 17 22:17:24.500699 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.500608    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9"
Jul 17 22:17:24.679094 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.678628    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9"]
Jul 17 22:17:24.686978 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:17:24.686946    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bec6b6c_93a1_41a5_b875_5152e7c0ce79.slice/crio-61513a3432eaf6dd910618f257007c4841e283e391bdceabae52a4ea2c9c90b1 WatchSource:0}: Error finding container 61513a3432eaf6dd910618f257007c4841e283e391bdceabae52a4ea2c9c90b1: Status 404 returned error can't find the container with id 61513a3432eaf6dd910618f257007c4841e283e391bdceabae52a4ea2c9c90b1
Jul 17 22:17:24.959948 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.959905    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5" event={"ID":"dfb4f8a6-8112-489a-8b02-a6e2b1f90c53","Type":"ContainerDied","Data":"be0f3e9eb70b0cc97f28606f25c9fa22f023ba473bbc011d83152e4a640c5e43"}
Jul 17 22:17:24.960176 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.959983    2565 scope.go:117] "RemoveContainer" containerID="c4ddef70d1d1eb26f505e8390b622d4d7b68089cb5cb7578ac1eee60d4c99613"
Jul 17 22:17:24.960176 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.960104    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5"
Jul 17 22:17:24.973798 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.973678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9" event={"ID":"8bec6b6c-93a1-41a5-b875-5152e7c0ce79","Type":"ContainerStarted","Data":"61513a3432eaf6dd910618f257007c4841e283e391bdceabae52a4ea2c9c90b1"}
Jul 17 22:17:24.993453 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.993419    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5"]
Jul 17 22:17:24.999501 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:24.999471    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-j8tm5"]
Jul 17 22:17:25.185098 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:25.185000    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfb4f8a6-8112-489a-8b02-a6e2b1f90c53" path="/var/lib/kubelet/pods/dfb4f8a6-8112-489a-8b02-a6e2b1f90c53/volumes"
Jul 17 22:17:25.983677 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:25.983577    2565 generic.go:356] "Generic (PLEG): container finished" podID="8bec6b6c-93a1-41a5-b875-5152e7c0ce79" containerID="a7fd730cafb736ed64baafe27e861dcf94576576224836e73547aa89ab0efc7e" exitCode=7
Jul 17 22:17:25.983888 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:25.983667    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9" event={"ID":"8bec6b6c-93a1-41a5-b875-5152e7c0ce79","Type":"ContainerDied","Data":"a7fd730cafb736ed64baafe27e861dcf94576576224836e73547aa89ab0efc7e"}
Jul 17 22:17:25.983984 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:25.983966    2565 scope.go:117] "RemoveContainer" containerID="a7fd730cafb736ed64baafe27e861dcf94576576224836e73547aa89ab0efc7e"
Jul 17 22:17:26.991572 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:26.991533    2565 generic.go:356] "Generic (PLEG): container finished" podID="8bec6b6c-93a1-41a5-b875-5152e7c0ce79" containerID="5dd4bd7c734bc5efec9ca155d6a6b230c77b6c9eaf23c4b84042fc02fc759e20" exitCode=7
Jul 17 22:17:26.992020 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:26.991625    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9" event={"ID":"8bec6b6c-93a1-41a5-b875-5152e7c0ce79","Type":"ContainerDied","Data":"5dd4bd7c734bc5efec9ca155d6a6b230c77b6c9eaf23c4b84042fc02fc759e20"}
Jul 17 22:17:26.992020 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:26.991675    2565 scope.go:117] "RemoveContainer" containerID="a7fd730cafb736ed64baafe27e861dcf94576576224836e73547aa89ab0efc7e"
Jul 17 22:17:26.992020 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:26.991932    2565 scope.go:117] "RemoveContainer" containerID="5dd4bd7c734bc5efec9ca155d6a6b230c77b6c9eaf23c4b84042fc02fc759e20"
Jul 17 22:17:26.992262 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:17:26.992234    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9_odh-ai-gateway-infra(8bec6b6c-93a1-41a5-b875-5152e7c0ce79)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9" podUID="8bec6b6c-93a1-41a5-b875-5152e7c0ce79"
Jul 17 22:17:27.999434 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:27.999407    2565 scope.go:117] "RemoveContainer" containerID="5dd4bd7c734bc5efec9ca155d6a6b230c77b6c9eaf23c4b84042fc02fc759e20"
Jul 17 22:17:27.999887 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:17:27.999639    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9_odh-ai-gateway-infra(8bec6b6c-93a1-41a5-b875-5152e7c0ce79)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9" podUID="8bec6b6c-93a1-41a5-b875-5152e7c0ce79"
Jul 17 22:17:30.012295 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:30.012256    2565 generic.go:356] "Generic (PLEG): container finished" podID="edcfbc0e-024d-41b6-a49d-f759bd3e53b6" containerID="30a35eea2ce7ca4bb75d11357bf6392ec8638133d63b3e2aaebf86ebc507247d" exitCode=0
Jul 17 22:17:30.012719 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:30.012324    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts" event={"ID":"edcfbc0e-024d-41b6-a49d-f759bd3e53b6","Type":"ContainerDied","Data":"30a35eea2ce7ca4bb75d11357bf6392ec8638133d63b3e2aaebf86ebc507247d"}
Jul 17 22:17:30.012762 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:30.012742    2565 scope.go:117] "RemoveContainer" containerID="30a35eea2ce7ca4bb75d11357bf6392ec8638133d63b3e2aaebf86ebc507247d"
Jul 17 22:17:30.013753 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:30.013727    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:17:30.013824 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:30.013808    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:17:30.013868 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:30.013855    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:17:31.019480 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:31.019444    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts" event={"ID":"edcfbc0e-024d-41b6-a49d-f759bd3e53b6","Type":"ContainerStarted","Data":"94414cd074013d0ae5c685dca83e4ce82d672b418fc90c1a592b5d4b18453038"}
Jul 17 22:17:31.182812 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:31.182770    2565 patch_prober.go:28] interesting pod/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 17 22:17:31.183019 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:31.182858    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts" podUID="edcfbc0e-024d-41b6-a49d-f759bd3e53b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 17 22:17:31.184597 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:31.184576    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:32.182036 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:32.181989    2565 patch_prober.go:28] interesting pod/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 17 22:17:32.182487 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:32.182092    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts" podUID="edcfbc0e-024d-41b6-a49d-f759bd3e53b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 17 22:17:33.181691 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:33.181651    2565 patch_prober.go:28] interesting pod/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 17 22:17:33.181924 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:33.181716    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts" podUID="edcfbc0e-024d-41b6-a49d-f759bd3e53b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 17 22:17:34.182212 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:34.182172    2565 patch_prober.go:28] interesting pod/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 17 22:17:34.182620 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:34.182234    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts" podUID="edcfbc0e-024d-41b6-a49d-f759bd3e53b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 17 22:17:34.927432 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:34.927399    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:35.177175 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:35.177143    2565 scope.go:117] "RemoveContainer" containerID="ef60acfdbe8b5db3638b2919338e8d62ab7b023f8d1bba5c81e491eb6c270661"
Jul 17 22:17:35.185960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:35.185931    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:35.187038 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:35.187017    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:17:36.046091 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:36.046036    2565 generic.go:356] "Generic (PLEG): container finished" podID="18a79ff6-639a-4aa4-b320-821f5af64ab9" containerID="b9212e80cf2626377788b04f908d5e6936f38abd26bde62d7ba8e5b0b69bb978" exitCode=6
Jul 17 22:17:36.046277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:36.046116    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5" event={"ID":"18a79ff6-639a-4aa4-b320-821f5af64ab9","Type":"ContainerDied","Data":"b9212e80cf2626377788b04f908d5e6936f38abd26bde62d7ba8e5b0b69bb978"}
Jul 17 22:17:36.046277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:36.046181    2565 scope.go:117] "RemoveContainer" containerID="ef60acfdbe8b5db3638b2919338e8d62ab7b023f8d1bba5c81e491eb6c270661"
Jul 17 22:17:36.046534 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:36.046514    2565 scope.go:117] "RemoveContainer" containerID="b9212e80cf2626377788b04f908d5e6936f38abd26bde62d7ba8e5b0b69bb978"
Jul 17 22:17:36.046747 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:17:36.046728    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5_odh-ai-gateway-infra(18a79ff6-639a-4aa4-b320-821f5af64ab9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5" podUID="18a79ff6-639a-4aa4-b320-821f5af64ab9"
Jul 17 22:17:37.072287 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:37.072254    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5"]
Jul 17 22:17:37.219748 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:37.219721    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5"
Jul 17 22:17:37.281196 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:37.281164    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a79ff6-639a-4aa4-b320-821f5af64ab9-maas-api-service-ca\") pod \"18a79ff6-639a-4aa4-b320-821f5af64ab9\" (UID: \"18a79ff6-639a-4aa4-b320-821f5af64ab9\") "
Jul 17 22:17:37.281595 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:37.281563    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a79ff6-639a-4aa4-b320-821f5af64ab9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18a79ff6-639a-4aa4-b320-821f5af64ab9" (UID: "18a79ff6-639a-4aa4-b320-821f5af64ab9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:17:37.382594 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:37.382505    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a79ff6-639a-4aa4-b320-821f5af64ab9-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:17:38.059488 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:38.059446    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5" event={"ID":"18a79ff6-639a-4aa4-b320-821f5af64ab9","Type":"ContainerDied","Data":"dbe8143bc15550964e89ec58e6b71a93b3f31e1647fd0dc1b0fa918a75eb4b69"}
Jul 17 22:17:38.059679 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:38.059501    2565 scope.go:117] "RemoveContainer" containerID="b9212e80cf2626377788b04f908d5e6936f38abd26bde62d7ba8e5b0b69bb978"
Jul 17 22:17:38.059679 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:38.059459    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5"
Jul 17 22:17:38.083872 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:38.083825    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5"]
Jul 17 22:17:38.091190 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:38.091164    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-t22k5"]
Jul 17 22:17:39.182003 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:39.181967    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18a79ff6-639a-4aa4-b320-821f5af64ab9" path="/var/lib/kubelet/pods/18a79ff6-639a-4aa4-b320-821f5af64ab9/volumes"
Jul 17 22:17:43.177344 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:43.177314    2565 scope.go:117] "RemoveContainer" containerID="5dd4bd7c734bc5efec9ca155d6a6b230c77b6c9eaf23c4b84042fc02fc759e20"
Jul 17 22:17:44.091000 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:44.090909    2565 generic.go:356] "Generic (PLEG): container finished" podID="8bec6b6c-93a1-41a5-b875-5152e7c0ce79" containerID="c702681c8d3f8b19a57c1195843b901719d2e47dd981ef4c9e79a1dca9a1e928" exitCode=7
Jul 17 22:17:44.091000 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:44.090987    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9" event={"ID":"8bec6b6c-93a1-41a5-b875-5152e7c0ce79","Type":"ContainerDied","Data":"c702681c8d3f8b19a57c1195843b901719d2e47dd981ef4c9e79a1dca9a1e928"}
Jul 17 22:17:44.091266 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:44.091061    2565 scope.go:117] "RemoveContainer" containerID="5dd4bd7c734bc5efec9ca155d6a6b230c77b6c9eaf23c4b84042fc02fc759e20"
Jul 17 22:17:44.091383 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:44.091369    2565 scope.go:117] "RemoveContainer" containerID="c702681c8d3f8b19a57c1195843b901719d2e47dd981ef4c9e79a1dca9a1e928"
Jul 17 22:17:44.091607 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:17:44.091588    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9_odh-ai-gateway-infra(8bec6b6c-93a1-41a5-b875-5152e7c0ce79)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9" podUID="8bec6b6c-93a1-41a5-b875-5152e7c0ce79"
Jul 17 22:17:45.120438 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:45.120399    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9"]
Jul 17 22:17:45.259850 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:45.259822    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9"
Jul 17 22:17:45.356272 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:45.356242    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bec6b6c-93a1-41a5-b875-5152e7c0ce79-maas-api-service-ca\") pod \"8bec6b6c-93a1-41a5-b875-5152e7c0ce79\" (UID: \"8bec6b6c-93a1-41a5-b875-5152e7c0ce79\") "
Jul 17 22:17:45.356616 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:45.356592    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bec6b6c-93a1-41a5-b875-5152e7c0ce79-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8bec6b6c-93a1-41a5-b875-5152e7c0ce79" (UID: "8bec6b6c-93a1-41a5-b875-5152e7c0ce79"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:17:45.458003 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:45.457909    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bec6b6c-93a1-41a5-b875-5152e7c0ce79-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:17:46.103871 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:46.103833    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9" event={"ID":"8bec6b6c-93a1-41a5-b875-5152e7c0ce79","Type":"ContainerDied","Data":"61513a3432eaf6dd910618f257007c4841e283e391bdceabae52a4ea2c9c90b1"}
Jul 17 22:17:46.104039 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:46.103893    2565 scope.go:117] "RemoveContainer" containerID="c702681c8d3f8b19a57c1195843b901719d2e47dd981ef4c9e79a1dca9a1e928"
Jul 17 22:17:46.104039 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:46.103847    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9"
Jul 17 22:17:46.125518 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:46.125482    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9"]
Jul 17 22:17:46.130538 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:46.130511    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-g8fs9"]
Jul 17 22:17:47.182838 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:47.182805    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bec6b6c-93a1-41a5-b875-5152e7c0ce79" path="/var/lib/kubelet/pods/8bec6b6c-93a1-41a5-b875-5152e7c0ce79/volumes"
Jul 17 22:17:55.690044 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.689995    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2"]
Jul 17 22:17:55.690566 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690543    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bec6b6c-93a1-41a5-b875-5152e7c0ce79" containerName="revoke-keys"
Jul 17 22:17:55.690610 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690571    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bec6b6c-93a1-41a5-b875-5152e7c0ce79" containerName="revoke-keys"
Jul 17 22:17:55.690610 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690587    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfb4f8a6-8112-489a-8b02-a6e2b1f90c53" containerName="revoke-keys"
Jul 17 22:17:55.690610 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690596    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfb4f8a6-8112-489a-8b02-a6e2b1f90c53" containerName="revoke-keys"
Jul 17 22:17:55.690701 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690612    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a79ff6-639a-4aa4-b320-821f5af64ab9" containerName="revoke-keys"
Jul 17 22:17:55.690701 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690622    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a79ff6-639a-4aa4-b320-821f5af64ab9" containerName="revoke-keys"
Jul 17 22:17:55.690701 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690634    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a79ff6-639a-4aa4-b320-821f5af64ab9" containerName="revoke-keys"
Jul 17 22:17:55.690701 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690641    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a79ff6-639a-4aa4-b320-821f5af64ab9" containerName="revoke-keys"
Jul 17 22:17:55.690701 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690657    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a79ff6-639a-4aa4-b320-821f5af64ab9" containerName="revoke-keys"
Jul 17 22:17:55.690701 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690666    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a79ff6-639a-4aa4-b320-821f5af64ab9" containerName="revoke-keys"
Jul 17 22:17:55.690701 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690682    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bec6b6c-93a1-41a5-b875-5152e7c0ce79" containerName="revoke-keys"
Jul 17 22:17:55.690701 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690691    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bec6b6c-93a1-41a5-b875-5152e7c0ce79" containerName="revoke-keys"
Jul 17 22:17:55.690942 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690720    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bec6b6c-93a1-41a5-b875-5152e7c0ce79" containerName="revoke-keys"
Jul 17 22:17:55.690942 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690729    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bec6b6c-93a1-41a5-b875-5152e7c0ce79" containerName="revoke-keys"
Jul 17 22:17:55.690942 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690811    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bec6b6c-93a1-41a5-b875-5152e7c0ce79" containerName="revoke-keys"
Jul 17 22:17:55.690942 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690823    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a79ff6-639a-4aa4-b320-821f5af64ab9" containerName="revoke-keys"
Jul 17 22:17:55.690942 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690833    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a79ff6-639a-4aa4-b320-821f5af64ab9" containerName="revoke-keys"
Jul 17 22:17:55.690942 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690841    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a79ff6-639a-4aa4-b320-821f5af64ab9" containerName="revoke-keys"
Jul 17 22:17:55.690942 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690853    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfb4f8a6-8112-489a-8b02-a6e2b1f90c53" containerName="revoke-keys"
Jul 17 22:17:55.690942 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.690861    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bec6b6c-93a1-41a5-b875-5152e7c0ce79" containerName="revoke-keys"
Jul 17 22:17:55.695262 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.695247    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2"
Jul 17 22:17:55.698432 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.698406    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:17:55.698601 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.698406    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:17:55.699863 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.699838    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2"]
Jul 17 22:17:55.750891 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.750850    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef036f4f-ceb4-4782-b884-9c0acfda24bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2\" (UID: \"ef036f4f-ceb4-4782-b884-9c0acfda24bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2"
Jul 17 22:17:55.851665 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.851633    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef036f4f-ceb4-4782-b884-9c0acfda24bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2\" (UID: \"ef036f4f-ceb4-4782-b884-9c0acfda24bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2"
Jul 17 22:17:55.852302 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:55.852283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef036f4f-ceb4-4782-b884-9c0acfda24bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2\" (UID: \"ef036f4f-ceb4-4782-b884-9c0acfda24bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2"
Jul 17 22:17:56.006768 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:56.006727    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2"
Jul 17 22:17:56.167963 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:56.167939    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2"]
Jul 17 22:17:56.170088 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:17:56.170037    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef036f4f_ceb4_4782_b884_9c0acfda24bc.slice/crio-bc6810cd81a24ef8f2671af3d01f478d745eecced33fa58814d0ac855965f4ab WatchSource:0}: Error finding container bc6810cd81a24ef8f2671af3d01f478d745eecced33fa58814d0ac855965f4ab: Status 404 returned error can't find the container with id bc6810cd81a24ef8f2671af3d01f478d745eecced33fa58814d0ac855965f4ab
Jul 17 22:17:57.156837 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:57.156740    2565 generic.go:356] "Generic (PLEG): container finished" podID="ef036f4f-ceb4-4782-b884-9c0acfda24bc" containerID="4536c7f95713fbb9c38e8b20f3246820cac5ad423b2b6c2f53d5fbe5ce7d5e3c" exitCode=6
Jul 17 22:17:57.156837 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:57.156826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2" event={"ID":"ef036f4f-ceb4-4782-b884-9c0acfda24bc","Type":"ContainerDied","Data":"4536c7f95713fbb9c38e8b20f3246820cac5ad423b2b6c2f53d5fbe5ce7d5e3c"}
Jul 17 22:17:57.157262 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:57.156876    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2" event={"ID":"ef036f4f-ceb4-4782-b884-9c0acfda24bc","Type":"ContainerStarted","Data":"bc6810cd81a24ef8f2671af3d01f478d745eecced33fa58814d0ac855965f4ab"}
Jul 17 22:17:57.157262 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:57.157018    2565 scope.go:117] "RemoveContainer" containerID="4536c7f95713fbb9c38e8b20f3246820cac5ad423b2b6c2f53d5fbe5ce7d5e3c"
Jul 17 22:17:58.164528 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:58.164441    2565 generic.go:356] "Generic (PLEG): container finished" podID="ef036f4f-ceb4-4782-b884-9c0acfda24bc" containerID="eb8f144b91a32df1cfb4e3d6c4cc80c3a7d268c15e6c89d597a82a51806de9df" exitCode=6
Jul 17 22:17:58.164528 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:58.164481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2" event={"ID":"ef036f4f-ceb4-4782-b884-9c0acfda24bc","Type":"ContainerDied","Data":"eb8f144b91a32df1cfb4e3d6c4cc80c3a7d268c15e6c89d597a82a51806de9df"}
Jul 17 22:17:58.164958 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:58.164569    2565 scope.go:117] "RemoveContainer" containerID="4536c7f95713fbb9c38e8b20f3246820cac5ad423b2b6c2f53d5fbe5ce7d5e3c"
Jul 17 22:17:58.164958 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:58.164834    2565 scope.go:117] "RemoveContainer" containerID="eb8f144b91a32df1cfb4e3d6c4cc80c3a7d268c15e6c89d597a82a51806de9df"
Jul 17 22:17:58.165085 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:17:58.165046    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2_odh-ai-gateway-infra(ef036f4f-ceb4-4782-b884-9c0acfda24bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2" podUID="ef036f4f-ceb4-4782-b884-9c0acfda24bc"
Jul 17 22:17:59.171747 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:17:59.171722    2565 scope.go:117] "RemoveContainer" containerID="eb8f144b91a32df1cfb4e3d6c4cc80c3a7d268c15e6c89d597a82a51806de9df"
Jul 17 22:17:59.172154 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:17:59.171992    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2_odh-ai-gateway-infra(ef036f4f-ceb4-4782-b884-9c0acfda24bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2" podUID="ef036f4f-ceb4-4782-b884-9c0acfda24bc"
Jul 17 22:18:11.177176 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:11.177142    2565 scope.go:117] "RemoveContainer" containerID="eb8f144b91a32df1cfb4e3d6c4cc80c3a7d268c15e6c89d597a82a51806de9df"
Jul 17 22:18:12.238267 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:12.238175    2565 generic.go:356] "Generic (PLEG): container finished" podID="ef036f4f-ceb4-4782-b884-9c0acfda24bc" containerID="9c2d6d2ebfa1b15426451e334ff23e1731db1fe23e55cd9a56eba3a8e3f172db" exitCode=6
Jul 17 22:18:12.238267 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:12.238252    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2" event={"ID":"ef036f4f-ceb4-4782-b884-9c0acfda24bc","Type":"ContainerDied","Data":"9c2d6d2ebfa1b15426451e334ff23e1731db1fe23e55cd9a56eba3a8e3f172db"}
Jul 17 22:18:12.238809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:12.238330    2565 scope.go:117] "RemoveContainer" containerID="eb8f144b91a32df1cfb4e3d6c4cc80c3a7d268c15e6c89d597a82a51806de9df"
Jul 17 22:18:12.238809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:12.238710    2565 scope.go:117] "RemoveContainer" containerID="9c2d6d2ebfa1b15426451e334ff23e1731db1fe23e55cd9a56eba3a8e3f172db"
Jul 17 22:18:12.238938 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:18:12.238909    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2_odh-ai-gateway-infra(ef036f4f-ceb4-4782-b884-9c0acfda24bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2" podUID="ef036f4f-ceb4-4782-b884-9c0acfda24bc"
Jul 17 22:18:13.268890 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:13.268852    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2"]
Jul 17 22:18:13.416470 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:13.416443    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2"
Jul 17 22:18:13.510291 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:13.510260    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef036f4f-ceb4-4782-b884-9c0acfda24bc-maas-api-service-ca\") pod \"ef036f4f-ceb4-4782-b884-9c0acfda24bc\" (UID: \"ef036f4f-ceb4-4782-b884-9c0acfda24bc\") "
Jul 17 22:18:13.510676 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:13.510648    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef036f4f-ceb4-4782-b884-9c0acfda24bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef036f4f-ceb4-4782-b884-9c0acfda24bc" (UID: "ef036f4f-ceb4-4782-b884-9c0acfda24bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:18:13.611049 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:13.610967    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef036f4f-ceb4-4782-b884-9c0acfda24bc-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:18:14.251615 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:14.251582    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2"
Jul 17 22:18:14.251820 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:14.251607    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2" event={"ID":"ef036f4f-ceb4-4782-b884-9c0acfda24bc","Type":"ContainerDied","Data":"bc6810cd81a24ef8f2671af3d01f478d745eecced33fa58814d0ac855965f4ab"}
Jul 17 22:18:14.251820 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:14.251691    2565 scope.go:117] "RemoveContainer" containerID="9c2d6d2ebfa1b15426451e334ff23e1731db1fe23e55cd9a56eba3a8e3f172db"
Jul 17 22:18:14.272833 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:14.272802    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2"]
Jul 17 22:18:14.278844 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:14.278816    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8qv2"]
Jul 17 22:18:15.182591 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:15.182555    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef036f4f-ceb4-4782-b884-9c0acfda24bc" path="/var/lib/kubelet/pods/ef036f4f-ceb4-4782-b884-9c0acfda24bc/volumes"
Jul 17 22:18:17.552403 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.552368    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr"]
Jul 17 22:18:17.552792 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.552732    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef036f4f-ceb4-4782-b884-9c0acfda24bc" containerName="revoke-keys"
Jul 17 22:18:17.552792 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.552744    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef036f4f-ceb4-4782-b884-9c0acfda24bc" containerName="revoke-keys"
Jul 17 22:18:17.552792 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.552758    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef036f4f-ceb4-4782-b884-9c0acfda24bc" containerName="revoke-keys"
Jul 17 22:18:17.552792 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.552763    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef036f4f-ceb4-4782-b884-9c0acfda24bc" containerName="revoke-keys"
Jul 17 22:18:17.552792 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.552768    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef036f4f-ceb4-4782-b884-9c0acfda24bc" containerName="revoke-keys"
Jul 17 22:18:17.552792 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.552773    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef036f4f-ceb4-4782-b884-9c0acfda24bc" containerName="revoke-keys"
Jul 17 22:18:17.552998 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.552824    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef036f4f-ceb4-4782-b884-9c0acfda24bc" containerName="revoke-keys"
Jul 17 22:18:17.552998 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.552833    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bec6b6c-93a1-41a5-b875-5152e7c0ce79" containerName="revoke-keys"
Jul 17 22:18:17.552998 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.552840    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef036f4f-ceb4-4782-b884-9c0acfda24bc" containerName="revoke-keys"
Jul 17 22:18:17.557366 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.557347    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr"
Jul 17 22:18:17.560297 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.560271    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:18:17.560435 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.560314    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:18:17.575535 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.575505    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr"]
Jul 17 22:18:17.647924 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.647875    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149766cf-2014-430a-b6f5-f752ed229ad5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr\" (UID: \"149766cf-2014-430a-b6f5-f752ed229ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr"
Jul 17 22:18:17.748900 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.748866    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149766cf-2014-430a-b6f5-f752ed229ad5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr\" (UID: \"149766cf-2014-430a-b6f5-f752ed229ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr"
Jul 17 22:18:17.749538 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.749517    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149766cf-2014-430a-b6f5-f752ed229ad5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr\" (UID: \"149766cf-2014-430a-b6f5-f752ed229ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr"
Jul 17 22:18:17.867828 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:17.867733    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr"
Jul 17 22:18:18.212547 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:18.212522    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr"]
Jul 17 22:18:18.214668 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:18:18.214635    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149766cf_2014_430a_b6f5_f752ed229ad5.slice/crio-5a7a0fe63ff37cee474a2e9ebca4b1e508c67ee32d8a2c4a58624d8c429e371b WatchSource:0}: Error finding container 5a7a0fe63ff37cee474a2e9ebca4b1e508c67ee32d8a2c4a58624d8c429e371b: Status 404 returned error can't find the container with id 5a7a0fe63ff37cee474a2e9ebca4b1e508c67ee32d8a2c4a58624d8c429e371b
Jul 17 22:18:18.273446 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:18.273411    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr" event={"ID":"149766cf-2014-430a-b6f5-f752ed229ad5","Type":"ContainerStarted","Data":"5a7a0fe63ff37cee474a2e9ebca4b1e508c67ee32d8a2c4a58624d8c429e371b"}
Jul 17 22:18:19.280320 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:19.280284    2565 generic.go:356] "Generic (PLEG): container finished" podID="149766cf-2014-430a-b6f5-f752ed229ad5" containerID="da222e144d69888eeac1b6962067aaf1dc2dfd78a241c41f65836fb79c753368" exitCode=6
Jul 17 22:18:19.280706 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:19.280376    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr" event={"ID":"149766cf-2014-430a-b6f5-f752ed229ad5","Type":"ContainerDied","Data":"da222e144d69888eeac1b6962067aaf1dc2dfd78a241c41f65836fb79c753368"}
Jul 17 22:18:19.280706 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:19.280512    2565 scope.go:117] "RemoveContainer" containerID="da222e144d69888eeac1b6962067aaf1dc2dfd78a241c41f65836fb79c753368"
Jul 17 22:18:20.293535 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:20.293501    2565 generic.go:356] "Generic (PLEG): container finished" podID="149766cf-2014-430a-b6f5-f752ed229ad5" containerID="a1ba42e1a4770c1cdb452f38f0c85ba5f6c41160c1384bfc5a448ad05a0ee020" exitCode=6
Jul 17 22:18:20.293972 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:20.293595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr" event={"ID":"149766cf-2014-430a-b6f5-f752ed229ad5","Type":"ContainerDied","Data":"a1ba42e1a4770c1cdb452f38f0c85ba5f6c41160c1384bfc5a448ad05a0ee020"}
Jul 17 22:18:20.293972 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:20.293707    2565 scope.go:117] "RemoveContainer" containerID="da222e144d69888eeac1b6962067aaf1dc2dfd78a241c41f65836fb79c753368"
Jul 17 22:18:20.293972 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:20.293965    2565 scope.go:117] "RemoveContainer" containerID="a1ba42e1a4770c1cdb452f38f0c85ba5f6c41160c1384bfc5a448ad05a0ee020"
Jul 17 22:18:20.294255 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:18:20.294234    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr_odh-ai-gateway-infra(149766cf-2014-430a-b6f5-f752ed229ad5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr" podUID="149766cf-2014-430a-b6f5-f752ed229ad5"
Jul 17 22:18:21.300767 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:21.300739    2565 scope.go:117] "RemoveContainer" containerID="a1ba42e1a4770c1cdb452f38f0c85ba5f6c41160c1384bfc5a448ad05a0ee020"
Jul 17 22:18:21.301172 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:18:21.300952    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr_odh-ai-gateway-infra(149766cf-2014-430a-b6f5-f752ed229ad5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr" podUID="149766cf-2014-430a-b6f5-f752ed229ad5"
Jul 17 22:18:24.207246 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:24.207208    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx"]
Jul 17 22:18:24.207705 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:24.207615    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef036f4f-ceb4-4782-b884-9c0acfda24bc" containerName="revoke-keys"
Jul 17 22:18:24.211588 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:24.211570    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx"
Jul 17 22:18:24.218346 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:24.218318    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx"]
Jul 17 22:18:24.311038 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:24.311003    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f91236ba-a8dd-4596-a301-7e47fd06d67a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx\" (UID: \"f91236ba-a8dd-4596-a301-7e47fd06d67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx"
Jul 17 22:18:24.412296 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:24.412250    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f91236ba-a8dd-4596-a301-7e47fd06d67a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx\" (UID: \"f91236ba-a8dd-4596-a301-7e47fd06d67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx"
Jul 17 22:18:24.412890 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:24.412870    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f91236ba-a8dd-4596-a301-7e47fd06d67a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx\" (UID: \"f91236ba-a8dd-4596-a301-7e47fd06d67a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx"
Jul 17 22:18:24.523442 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:24.523397    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx"
Jul 17 22:18:24.659655 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:24.659629    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx"]
Jul 17 22:18:24.662162 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:18:24.662136    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91236ba_a8dd_4596_a301_7e47fd06d67a.slice/crio-2d0c303d390bfbd8971ccc493b4415583b1b9589d859a9a58ac713588807f845 WatchSource:0}: Error finding container 2d0c303d390bfbd8971ccc493b4415583b1b9589d859a9a58ac713588807f845: Status 404 returned error can't find the container with id 2d0c303d390bfbd8971ccc493b4415583b1b9589d859a9a58ac713588807f845
Jul 17 22:18:25.322304 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:25.322265    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx" event={"ID":"f91236ba-a8dd-4596-a301-7e47fd06d67a","Type":"ContainerStarted","Data":"f19eac255d95dc6cf4d66d0299f84a70355bc5573c190086117c8f6de0fd7be5"}
Jul 17 22:18:25.322777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:25.322323    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx" event={"ID":"f91236ba-a8dd-4596-a301-7e47fd06d67a","Type":"ContainerStarted","Data":"2d0c303d390bfbd8971ccc493b4415583b1b9589d859a9a58ac713588807f845"}
Jul 17 22:18:25.342804 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:25.342756    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx" podStartSLOduration=1.342738911 podStartE2EDuration="1.342738911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:18:25.342451232 +0000 UTC m=+1968.731467883" watchObservedRunningTime="2026-07-17 22:18:25.342738911 +0000 UTC m=+1968.731755562"
Jul 17 22:18:26.328180 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:26.328142    2565 generic.go:356] "Generic (PLEG): container finished" podID="f91236ba-a8dd-4596-a301-7e47fd06d67a" containerID="f19eac255d95dc6cf4d66d0299f84a70355bc5573c190086117c8f6de0fd7be5" exitCode=7
Jul 17 22:18:26.328557 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:26.328197    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx" event={"ID":"f91236ba-a8dd-4596-a301-7e47fd06d67a","Type":"ContainerDied","Data":"f19eac255d95dc6cf4d66d0299f84a70355bc5573c190086117c8f6de0fd7be5"}
Jul 17 22:18:26.328557 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:26.328409    2565 scope.go:117] "RemoveContainer" containerID="f19eac255d95dc6cf4d66d0299f84a70355bc5573c190086117c8f6de0fd7be5"
Jul 17 22:18:27.335189 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:27.335092    2565 generic.go:356] "Generic (PLEG): container finished" podID="f91236ba-a8dd-4596-a301-7e47fd06d67a" containerID="8930d30558ffff839569c890c3d578eacc7210d99fe3f908608707605a2ac790" exitCode=7
Jul 17 22:18:27.335189 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:27.335168    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx" event={"ID":"f91236ba-a8dd-4596-a301-7e47fd06d67a","Type":"ContainerDied","Data":"8930d30558ffff839569c890c3d578eacc7210d99fe3f908608707605a2ac790"}
Jul 17 22:18:27.335691 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:27.335230    2565 scope.go:117] "RemoveContainer" containerID="f19eac255d95dc6cf4d66d0299f84a70355bc5573c190086117c8f6de0fd7be5"
Jul 17 22:18:27.335691 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:27.335494    2565 scope.go:117] "RemoveContainer" containerID="8930d30558ffff839569c890c3d578eacc7210d99fe3f908608707605a2ac790"
Jul 17 22:18:27.335796 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:18:27.335743    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx_odh-ai-gateway-infra(f91236ba-a8dd-4596-a301-7e47fd06d67a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx" podUID="f91236ba-a8dd-4596-a301-7e47fd06d67a"
Jul 17 22:18:28.341649 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:28.341578    2565 scope.go:117] "RemoveContainer" containerID="8930d30558ffff839569c890c3d578eacc7210d99fe3f908608707605a2ac790"
Jul 17 22:18:28.342019 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:18:28.341781    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx_odh-ai-gateway-infra(f91236ba-a8dd-4596-a301-7e47fd06d67a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx" podUID="f91236ba-a8dd-4596-a301-7e47fd06d67a"
Jul 17 22:18:34.176681 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:34.176646    2565 scope.go:117] "RemoveContainer" containerID="a1ba42e1a4770c1cdb452f38f0c85ba5f6c41160c1384bfc5a448ad05a0ee020"
Jul 17 22:18:35.377208 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:35.377172    2565 generic.go:356] "Generic (PLEG): container finished" podID="149766cf-2014-430a-b6f5-f752ed229ad5" containerID="0a6ac0afacccbd462e80ad7630c30eae324c89d99090939ea36d001a41928c78" exitCode=6
Jul 17 22:18:35.377674 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:35.377248    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr" event={"ID":"149766cf-2014-430a-b6f5-f752ed229ad5","Type":"ContainerDied","Data":"0a6ac0afacccbd462e80ad7630c30eae324c89d99090939ea36d001a41928c78"}
Jul 17 22:18:35.377674 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:35.377314    2565 scope.go:117] "RemoveContainer" containerID="a1ba42e1a4770c1cdb452f38f0c85ba5f6c41160c1384bfc5a448ad05a0ee020"
Jul 17 22:18:35.377674 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:35.377593    2565 scope.go:117] "RemoveContainer" containerID="0a6ac0afacccbd462e80ad7630c30eae324c89d99090939ea36d001a41928c78"
Jul 17 22:18:35.377846 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:18:35.377815    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr_odh-ai-gateway-infra(149766cf-2014-430a-b6f5-f752ed229ad5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr" podUID="149766cf-2014-430a-b6f5-f752ed229ad5"
Jul 17 22:18:36.402765 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:36.402728    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr"]
Jul 17 22:18:36.541345 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:36.541318    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr"
Jul 17 22:18:36.628556 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:36.628524    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149766cf-2014-430a-b6f5-f752ed229ad5-maas-api-service-ca\") pod \"149766cf-2014-430a-b6f5-f752ed229ad5\" (UID: \"149766cf-2014-430a-b6f5-f752ed229ad5\") "
Jul 17 22:18:36.628871 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:36.628850    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/149766cf-2014-430a-b6f5-f752ed229ad5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "149766cf-2014-430a-b6f5-f752ed229ad5" (UID: "149766cf-2014-430a-b6f5-f752ed229ad5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:18:36.729543 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:36.729454    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149766cf-2014-430a-b6f5-f752ed229ad5-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:18:37.389814 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:37.389785    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr"
Jul 17 22:18:37.390001 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:37.389785    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr" event={"ID":"149766cf-2014-430a-b6f5-f752ed229ad5","Type":"ContainerDied","Data":"5a7a0fe63ff37cee474a2e9ebca4b1e508c67ee32d8a2c4a58624d8c429e371b"}
Jul 17 22:18:37.390001 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:37.389919    2565 scope.go:117] "RemoveContainer" containerID="0a6ac0afacccbd462e80ad7630c30eae324c89d99090939ea36d001a41928c78"
Jul 17 22:18:37.407549 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:37.407506    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr"]
Jul 17 22:18:37.413904 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:37.413876    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2hvdr"]
Jul 17 22:18:39.182771 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:39.182734    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="149766cf-2014-430a-b6f5-f752ed229ad5" path="/var/lib/kubelet/pods/149766cf-2014-430a-b6f5-f752ed229ad5/volumes"
Jul 17 22:18:41.177299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:41.177267    2565 scope.go:117] "RemoveContainer" containerID="8930d30558ffff839569c890c3d578eacc7210d99fe3f908608707605a2ac790"
Jul 17 22:18:41.411298 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:41.411266    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx" event={"ID":"f91236ba-a8dd-4596-a301-7e47fd06d67a","Type":"ContainerStarted","Data":"1184ec9fc425b60d115510baf1315dd63b4c0d848e81600ec7d18c8d4e13cdbc"}
Jul 17 22:18:42.418705 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:42.418674    2565 generic.go:356] "Generic (PLEG): container finished" podID="f91236ba-a8dd-4596-a301-7e47fd06d67a" containerID="1184ec9fc425b60d115510baf1315dd63b4c0d848e81600ec7d18c8d4e13cdbc" exitCode=7
Jul 17 22:18:42.419187 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:42.418750    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx" event={"ID":"f91236ba-a8dd-4596-a301-7e47fd06d67a","Type":"ContainerDied","Data":"1184ec9fc425b60d115510baf1315dd63b4c0d848e81600ec7d18c8d4e13cdbc"}
Jul 17 22:18:42.419187 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:42.418824    2565 scope.go:117] "RemoveContainer" containerID="8930d30558ffff839569c890c3d578eacc7210d99fe3f908608707605a2ac790"
Jul 17 22:18:42.419341 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:42.419236    2565 scope.go:117] "RemoveContainer" containerID="1184ec9fc425b60d115510baf1315dd63b4c0d848e81600ec7d18c8d4e13cdbc"
Jul 17 22:18:42.419476 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:18:42.419453    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx_odh-ai-gateway-infra(f91236ba-a8dd-4596-a301-7e47fd06d67a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx" podUID="f91236ba-a8dd-4596-a301-7e47fd06d67a"
Jul 17 22:18:42.445455 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:42.445416    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx"]
Jul 17 22:18:43.567359 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:43.567330    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx"
Jul 17 22:18:43.592217 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:43.592191    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f91236ba-a8dd-4596-a301-7e47fd06d67a-maas-api-service-ca\") pod \"f91236ba-a8dd-4596-a301-7e47fd06d67a\" (UID: \"f91236ba-a8dd-4596-a301-7e47fd06d67a\") "
Jul 17 22:18:43.592523 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:43.592503    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91236ba-a8dd-4596-a301-7e47fd06d67a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f91236ba-a8dd-4596-a301-7e47fd06d67a" (UID: "f91236ba-a8dd-4596-a301-7e47fd06d67a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:18:43.693786 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:43.693689    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f91236ba-a8dd-4596-a301-7e47fd06d67a-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:18:44.431583 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:44.431550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx" event={"ID":"f91236ba-a8dd-4596-a301-7e47fd06d67a","Type":"ContainerDied","Data":"2d0c303d390bfbd8971ccc493b4415583b1b9589d859a9a58ac713588807f845"}
Jul 17 22:18:44.431583 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:44.431576    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx"
Jul 17 22:18:44.431830 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:44.431635    2565 scope.go:117] "RemoveContainer" containerID="1184ec9fc425b60d115510baf1315dd63b4c0d848e81600ec7d18c8d4e13cdbc"
Jul 17 22:18:44.454717 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:44.454686    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx"]
Jul 17 22:18:44.460233 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:44.460213    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mcdsx"]
Jul 17 22:18:45.182495 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.182457    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f91236ba-a8dd-4596-a301-7e47fd06d67a" path="/var/lib/kubelet/pods/f91236ba-a8dd-4596-a301-7e47fd06d67a/volumes"
Jul 17 22:18:45.746948 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.746913    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv"]
Jul 17 22:18:45.747296 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.747283    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f91236ba-a8dd-4596-a301-7e47fd06d67a" containerName="revoke-keys"
Jul 17 22:18:45.747352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.747298    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f91236ba-a8dd-4596-a301-7e47fd06d67a" containerName="revoke-keys"
Jul 17 22:18:45.747352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.747308    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f91236ba-a8dd-4596-a301-7e47fd06d67a" containerName="revoke-keys"
Jul 17 22:18:45.747352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.747314    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f91236ba-a8dd-4596-a301-7e47fd06d67a" containerName="revoke-keys"
Jul 17 22:18:45.747352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.747322    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f91236ba-a8dd-4596-a301-7e47fd06d67a" containerName="revoke-keys"
Jul 17 22:18:45.747352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.747327    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f91236ba-a8dd-4596-a301-7e47fd06d67a" containerName="revoke-keys"
Jul 17 22:18:45.747352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.747343    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="149766cf-2014-430a-b6f5-f752ed229ad5" containerName="revoke-keys"
Jul 17 22:18:45.747352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.747347    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="149766cf-2014-430a-b6f5-f752ed229ad5" containerName="revoke-keys"
Jul 17 22:18:45.747352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.747354    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="149766cf-2014-430a-b6f5-f752ed229ad5" containerName="revoke-keys"
Jul 17 22:18:45.747593 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.747360    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="149766cf-2014-430a-b6f5-f752ed229ad5" containerName="revoke-keys"
Jul 17 22:18:45.747593 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.747417    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="149766cf-2014-430a-b6f5-f752ed229ad5" containerName="revoke-keys"
Jul 17 22:18:45.747593 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.747424    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="149766cf-2014-430a-b6f5-f752ed229ad5" containerName="revoke-keys"
Jul 17 22:18:45.747593 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.747430    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f91236ba-a8dd-4596-a301-7e47fd06d67a" containerName="revoke-keys"
Jul 17 22:18:45.747593 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.747436    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f91236ba-a8dd-4596-a301-7e47fd06d67a" containerName="revoke-keys"
Jul 17 22:18:45.751977 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.751960    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv"
Jul 17 22:18:45.754632 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.754615    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:18:45.754713 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.754665    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:18:45.759478 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.758982    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv"]
Jul 17 22:18:45.814572 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.814532    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f983a1c0-2439-497d-9734-f14231d3574f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv\" (UID: \"f983a1c0-2439-497d-9734-f14231d3574f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv"
Jul 17 22:18:45.915152 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.915110    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f983a1c0-2439-497d-9734-f14231d3574f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv\" (UID: \"f983a1c0-2439-497d-9734-f14231d3574f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv"
Jul 17 22:18:45.915774 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:45.915749    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f983a1c0-2439-497d-9734-f14231d3574f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv\" (UID: \"f983a1c0-2439-497d-9734-f14231d3574f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv"
Jul 17 22:18:46.062839 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:46.062751    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv"
Jul 17 22:18:46.200400 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:46.200351    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv"]
Jul 17 22:18:46.203308 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:18:46.203282    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf983a1c0_2439_497d_9734_f14231d3574f.slice/crio-cb439a6145953c3eb3265edf11008f455d3743f862b22d3273bffbe0a0ec315d WatchSource:0}: Error finding container cb439a6145953c3eb3265edf11008f455d3743f862b22d3273bffbe0a0ec315d: Status 404 returned error can't find the container with id cb439a6145953c3eb3265edf11008f455d3743f862b22d3273bffbe0a0ec315d
Jul 17 22:18:46.443903 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:46.443866    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv" event={"ID":"f983a1c0-2439-497d-9734-f14231d3574f","Type":"ContainerStarted","Data":"cb439a6145953c3eb3265edf11008f455d3743f862b22d3273bffbe0a0ec315d"}
Jul 17 22:18:47.450696 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:47.450658    2565 generic.go:356] "Generic (PLEG): container finished" podID="f983a1c0-2439-497d-9734-f14231d3574f" containerID="14cb643174847f5d06bd93da8deaf9568426ff6593c9a7f29751ebc71ca5970a" exitCode=6
Jul 17 22:18:47.451113 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:47.450737    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv" event={"ID":"f983a1c0-2439-497d-9734-f14231d3574f","Type":"ContainerDied","Data":"14cb643174847f5d06bd93da8deaf9568426ff6593c9a7f29751ebc71ca5970a"}
Jul 17 22:18:47.451113 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:47.450892    2565 scope.go:117] "RemoveContainer" containerID="14cb643174847f5d06bd93da8deaf9568426ff6593c9a7f29751ebc71ca5970a"
Jul 17 22:18:48.457445 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:48.457351    2565 generic.go:356] "Generic (PLEG): container finished" podID="f983a1c0-2439-497d-9734-f14231d3574f" containerID="a8527cc34285b0a9f613fc9cd17a1a024ce2806f146913ec12d5b32534a47849" exitCode=6
Jul 17 22:18:48.457445 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:48.457387    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv" event={"ID":"f983a1c0-2439-497d-9734-f14231d3574f","Type":"ContainerDied","Data":"a8527cc34285b0a9f613fc9cd17a1a024ce2806f146913ec12d5b32534a47849"}
Jul 17 22:18:48.457951 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:48.457452    2565 scope.go:117] "RemoveContainer" containerID="14cb643174847f5d06bd93da8deaf9568426ff6593c9a7f29751ebc71ca5970a"
Jul 17 22:18:48.457951 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:48.457661    2565 scope.go:117] "RemoveContainer" containerID="a8527cc34285b0a9f613fc9cd17a1a024ce2806f146913ec12d5b32534a47849"
Jul 17 22:18:48.457951 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:18:48.457908    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv_odh-ai-gateway-infra(f983a1c0-2439-497d-9734-f14231d3574f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv" podUID="f983a1c0-2439-497d-9734-f14231d3574f"
Jul 17 22:18:49.463364 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:18:49.463332    2565 scope.go:117] "RemoveContainer" containerID="a8527cc34285b0a9f613fc9cd17a1a024ce2806f146913ec12d5b32534a47849"
Jul 17 22:18:49.463738 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:18:49.463530    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv_odh-ai-gateway-infra(f983a1c0-2439-497d-9734-f14231d3574f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv" podUID="f983a1c0-2439-497d-9734-f14231d3574f"
Jul 17 22:19:00.176709 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:00.176673    2565 scope.go:117] "RemoveContainer" containerID="a8527cc34285b0a9f613fc9cd17a1a024ce2806f146913ec12d5b32534a47849"
Jul 17 22:19:00.513896 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:00.513858    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv" event={"ID":"f983a1c0-2439-497d-9734-f14231d3574f","Type":"ContainerStarted","Data":"4f3575ffaa5cef466ede5a0bf29b0357799b7580a51aa3a1db81f01fdd141a03"}
Jul 17 22:19:00.530318 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:00.530265    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv" podStartSLOduration=15.530249906 podStartE2EDuration="15.530249906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:19:00.528406209 +0000 UTC m=+2003.917422861" watchObservedRunningTime="2026-07-17 22:19:00.530249906 +0000 UTC m=+2003.919266536"
Jul 17 22:19:01.520565 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:01.520533    2565 generic.go:356] "Generic (PLEG): container finished" podID="f983a1c0-2439-497d-9734-f14231d3574f" containerID="4f3575ffaa5cef466ede5a0bf29b0357799b7580a51aa3a1db81f01fdd141a03" exitCode=6
Jul 17 22:19:01.521051 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:01.520611    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv" event={"ID":"f983a1c0-2439-497d-9734-f14231d3574f","Type":"ContainerDied","Data":"4f3575ffaa5cef466ede5a0bf29b0357799b7580a51aa3a1db81f01fdd141a03"}
Jul 17 22:19:01.521051 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:01.520687    2565 scope.go:117] "RemoveContainer" containerID="a8527cc34285b0a9f613fc9cd17a1a024ce2806f146913ec12d5b32534a47849"
Jul 17 22:19:01.521051 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:01.520970    2565 scope.go:117] "RemoveContainer" containerID="4f3575ffaa5cef466ede5a0bf29b0357799b7580a51aa3a1db81f01fdd141a03"
Jul 17 22:19:01.521235 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:19:01.521213    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv_odh-ai-gateway-infra(f983a1c0-2439-497d-9734-f14231d3574f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv" podUID="f983a1c0-2439-497d-9734-f14231d3574f"
Jul 17 22:19:01.548793 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:01.548757    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv"]
Jul 17 22:19:02.666897 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:02.666870    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv"
Jul 17 22:19:02.781238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:02.781150    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f983a1c0-2439-497d-9734-f14231d3574f-maas-api-service-ca\") pod \"f983a1c0-2439-497d-9734-f14231d3574f\" (UID: \"f983a1c0-2439-497d-9734-f14231d3574f\") "
Jul 17 22:19:02.781508 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:02.781484    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f983a1c0-2439-497d-9734-f14231d3574f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f983a1c0-2439-497d-9734-f14231d3574f" (UID: "f983a1c0-2439-497d-9734-f14231d3574f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:19:02.882576 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:02.882533    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f983a1c0-2439-497d-9734-f14231d3574f-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:19:03.532198 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:03.532162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv" event={"ID":"f983a1c0-2439-497d-9734-f14231d3574f","Type":"ContainerDied","Data":"cb439a6145953c3eb3265edf11008f455d3743f862b22d3273bffbe0a0ec315d"}
Jul 17 22:19:03.532198 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:03.532196    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv"
Jul 17 22:19:03.532418 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:03.532256    2565 scope.go:117] "RemoveContainer" containerID="4f3575ffaa5cef466ede5a0bf29b0357799b7580a51aa3a1db81f01fdd141a03"
Jul 17 22:19:03.549907 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:03.549869    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv"]
Jul 17 22:19:03.560084 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:03.560041    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ctkqv"]
Jul 17 22:19:05.181826 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:05.181789    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f983a1c0-2439-497d-9734-f14231d3574f" path="/var/lib/kubelet/pods/f983a1c0-2439-497d-9734-f14231d3574f/volumes"
Jul 17 22:19:07.580271 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.580240    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv"]
Jul 17 22:19:07.580697 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.580593    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="149766cf-2014-430a-b6f5-f752ed229ad5" containerName="revoke-keys"
Jul 17 22:19:07.580697 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.580603    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="149766cf-2014-430a-b6f5-f752ed229ad5" containerName="revoke-keys"
Jul 17 22:19:07.580697 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.580612    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f983a1c0-2439-497d-9734-f14231d3574f" containerName="revoke-keys"
Jul 17 22:19:07.580697 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.580618    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f983a1c0-2439-497d-9734-f14231d3574f" containerName="revoke-keys"
Jul 17 22:19:07.580697 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.580628    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f983a1c0-2439-497d-9734-f14231d3574f" containerName="revoke-keys"
Jul 17 22:19:07.580697 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.580634    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f983a1c0-2439-497d-9734-f14231d3574f" containerName="revoke-keys"
Jul 17 22:19:07.580697 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.580693    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f983a1c0-2439-497d-9734-f14231d3574f" containerName="revoke-keys"
Jul 17 22:19:07.580697 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.580701    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="149766cf-2014-430a-b6f5-f752ed229ad5" containerName="revoke-keys"
Jul 17 22:19:07.580971 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.580708    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f91236ba-a8dd-4596-a301-7e47fd06d67a" containerName="revoke-keys"
Jul 17 22:19:07.580971 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.580714    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f983a1c0-2439-497d-9734-f14231d3574f" containerName="revoke-keys"
Jul 17 22:19:07.585017 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.584997    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv"
Jul 17 22:19:07.587813 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.587782    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:19:07.587953 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.587879    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:19:07.591281 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.591258    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv"]
Jul 17 22:19:07.725493 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.725454    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d01405-f62c-4b81-bd39-3464260e1fd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv\" (UID: \"d4d01405-f62c-4b81-bd39-3464260e1fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv"
Jul 17 22:19:07.826487 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.826447    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d01405-f62c-4b81-bd39-3464260e1fd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv\" (UID: \"d4d01405-f62c-4b81-bd39-3464260e1fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv"
Jul 17 22:19:07.827169 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.827145    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d01405-f62c-4b81-bd39-3464260e1fd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv\" (UID: \"d4d01405-f62c-4b81-bd39-3464260e1fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv"
Jul 17 22:19:07.898224 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:07.898122    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv"
Jul 17 22:19:08.041801 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:08.041774    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv"]
Jul 17 22:19:08.044598 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:19:08.044566    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d01405_f62c_4b81_bd39_3464260e1fd7.slice/crio-4ba42846132a42f188fff12da082432fb0fd9369419552d8e6afaf8b23beec55 WatchSource:0}: Error finding container 4ba42846132a42f188fff12da082432fb0fd9369419552d8e6afaf8b23beec55: Status 404 returned error can't find the container with id 4ba42846132a42f188fff12da082432fb0fd9369419552d8e6afaf8b23beec55
Jul 17 22:19:08.558808 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:08.558768    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv" event={"ID":"d4d01405-f62c-4b81-bd39-3464260e1fd7","Type":"ContainerStarted","Data":"3f8555f0c9958a191b280ddb380920144080d8e224f73d9e6dbd6a3ad33f2e9e"}
Jul 17 22:19:08.558970 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:08.558835    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv" event={"ID":"d4d01405-f62c-4b81-bd39-3464260e1fd7","Type":"ContainerStarted","Data":"4ba42846132a42f188fff12da082432fb0fd9369419552d8e6afaf8b23beec55"}
Jul 17 22:19:08.576230 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:08.576175    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv" podStartSLOduration=1.57615779 podStartE2EDuration="1.57615779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:19:08.574107291 +0000 UTC m=+2011.963123954" watchObservedRunningTime="2026-07-17 22:19:08.57615779 +0000 UTC m=+2011.965174441"
Jul 17 22:19:09.564703 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:09.564670    2565 generic.go:356] "Generic (PLEG): container finished" podID="d4d01405-f62c-4b81-bd39-3464260e1fd7" containerID="3f8555f0c9958a191b280ddb380920144080d8e224f73d9e6dbd6a3ad33f2e9e" exitCode=6
Jul 17 22:19:09.565109 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:09.564741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv" event={"ID":"d4d01405-f62c-4b81-bd39-3464260e1fd7","Type":"ContainerDied","Data":"3f8555f0c9958a191b280ddb380920144080d8e224f73d9e6dbd6a3ad33f2e9e"}
Jul 17 22:19:09.565109 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:09.564954    2565 scope.go:117] "RemoveContainer" containerID="3f8555f0c9958a191b280ddb380920144080d8e224f73d9e6dbd6a3ad33f2e9e"
Jul 17 22:19:10.571438 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:10.571396    2565 generic.go:356] "Generic (PLEG): container finished" podID="d4d01405-f62c-4b81-bd39-3464260e1fd7" containerID="9c4735c5f8556b34fffc293e1a6c7f2fea653a6131a2ec65c0e4424657b8a4b2" exitCode=6
Jul 17 22:19:10.571926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:10.571447    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv" event={"ID":"d4d01405-f62c-4b81-bd39-3464260e1fd7","Type":"ContainerDied","Data":"9c4735c5f8556b34fffc293e1a6c7f2fea653a6131a2ec65c0e4424657b8a4b2"}
Jul 17 22:19:10.571926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:10.571505    2565 scope.go:117] "RemoveContainer" containerID="3f8555f0c9958a191b280ddb380920144080d8e224f73d9e6dbd6a3ad33f2e9e"
Jul 17 22:19:10.571926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:10.571665    2565 scope.go:117] "RemoveContainer" containerID="9c4735c5f8556b34fffc293e1a6c7f2fea653a6131a2ec65c0e4424657b8a4b2"
Jul 17 22:19:10.572054 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:19:10.571926    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv_odh-ai-gateway-infra(d4d01405-f62c-4b81-bd39-3464260e1fd7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv" podUID="d4d01405-f62c-4b81-bd39-3464260e1fd7"
Jul 17 22:19:11.577856 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:11.577826    2565 scope.go:117] "RemoveContainer" containerID="9c4735c5f8556b34fffc293e1a6c7f2fea653a6131a2ec65c0e4424657b8a4b2"
Jul 17 22:19:11.578292 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:19:11.578035    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv_odh-ai-gateway-infra(d4d01405-f62c-4b81-bd39-3464260e1fd7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv" podUID="d4d01405-f62c-4b81-bd39-3464260e1fd7"
Jul 17 22:19:24.258691 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:24.258660    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh"]
Jul 17 22:19:24.259163 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:24.259061    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f983a1c0-2439-497d-9734-f14231d3574f" containerName="revoke-keys"
Jul 17 22:19:24.259163 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:24.259121    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f983a1c0-2439-497d-9734-f14231d3574f" containerName="revoke-keys"
Jul 17 22:19:24.259282 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:24.259223    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f983a1c0-2439-497d-9734-f14231d3574f" containerName="revoke-keys"
Jul 17 22:19:24.263392 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:24.263368    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh"
Jul 17 22:19:24.269576 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:24.269535    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh"]
Jul 17 22:19:24.387484 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:24.387444    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71894af1-cf6f-4960-85f7-5a8fbc942aae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh\" (UID: \"71894af1-cf6f-4960-85f7-5a8fbc942aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh"
Jul 17 22:19:24.488262 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:24.488204    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71894af1-cf6f-4960-85f7-5a8fbc942aae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh\" (UID: \"71894af1-cf6f-4960-85f7-5a8fbc942aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh"
Jul 17 22:19:24.488856 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:24.488838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71894af1-cf6f-4960-85f7-5a8fbc942aae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh\" (UID: \"71894af1-cf6f-4960-85f7-5a8fbc942aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh"
Jul 17 22:19:24.576954 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:24.576865    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh"
Jul 17 22:19:24.716003 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:24.715975    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh"]
Jul 17 22:19:24.718685 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:19:24.718656    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71894af1_cf6f_4960_85f7_5a8fbc942aae.slice/crio-9b783caaf46a796389b3bd3646f8c8ae333134a2d86e50f9f33d81cdf80438f5 WatchSource:0}: Error finding container 9b783caaf46a796389b3bd3646f8c8ae333134a2d86e50f9f33d81cdf80438f5: Status 404 returned error can't find the container with id 9b783caaf46a796389b3bd3646f8c8ae333134a2d86e50f9f33d81cdf80438f5
Jul 17 22:19:25.648005 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:25.647910    2565 generic.go:356] "Generic (PLEG): container finished" podID="71894af1-cf6f-4960-85f7-5a8fbc942aae" containerID="8752b922b25c287ccece636b995145a84b8bff66065940b0e3379522b65e0994" exitCode=7
Jul 17 22:19:25.648383 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:25.648008    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh" event={"ID":"71894af1-cf6f-4960-85f7-5a8fbc942aae","Type":"ContainerDied","Data":"8752b922b25c287ccece636b995145a84b8bff66065940b0e3379522b65e0994"}
Jul 17 22:19:25.648383 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:25.648060    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh" event={"ID":"71894af1-cf6f-4960-85f7-5a8fbc942aae","Type":"ContainerStarted","Data":"9b783caaf46a796389b3bd3646f8c8ae333134a2d86e50f9f33d81cdf80438f5"}
Jul 17 22:19:25.648383 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:25.648150    2565 scope.go:117] "RemoveContainer" containerID="8752b922b25c287ccece636b995145a84b8bff66065940b0e3379522b65e0994"
Jul 17 22:19:26.176676 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:26.176640    2565 scope.go:117] "RemoveContainer" containerID="9c4735c5f8556b34fffc293e1a6c7f2fea653a6131a2ec65c0e4424657b8a4b2"
Jul 17 22:19:26.654398 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:26.654308    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv" event={"ID":"d4d01405-f62c-4b81-bd39-3464260e1fd7","Type":"ContainerStarted","Data":"28633b642daac3ee2bc1591a17c09f8ac00425ff55177581258e4703f801942c"}
Jul 17 22:19:26.656012 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:26.655981    2565 generic.go:356] "Generic (PLEG): container finished" podID="71894af1-cf6f-4960-85f7-5a8fbc942aae" containerID="cd2e781d1c6b46f2f4b749d8349f18f66ac65e5d7dd68ed6a078976a7d06ab5e" exitCode=7
Jul 17 22:19:26.656152 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:26.656056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh" event={"ID":"71894af1-cf6f-4960-85f7-5a8fbc942aae","Type":"ContainerDied","Data":"cd2e781d1c6b46f2f4b749d8349f18f66ac65e5d7dd68ed6a078976a7d06ab5e"}
Jul 17 22:19:26.656152 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:26.656136    2565 scope.go:117] "RemoveContainer" containerID="8752b922b25c287ccece636b995145a84b8bff66065940b0e3379522b65e0994"
Jul 17 22:19:26.656369 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:26.656353    2565 scope.go:117] "RemoveContainer" containerID="cd2e781d1c6b46f2f4b749d8349f18f66ac65e5d7dd68ed6a078976a7d06ab5e"
Jul 17 22:19:26.656591 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:19:26.656568    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh_odh-ai-gateway-infra(71894af1-cf6f-4960-85f7-5a8fbc942aae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh" podUID="71894af1-cf6f-4960-85f7-5a8fbc942aae"
Jul 17 22:19:27.662395 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:27.662364    2565 generic.go:356] "Generic (PLEG): container finished" podID="d4d01405-f62c-4b81-bd39-3464260e1fd7" containerID="28633b642daac3ee2bc1591a17c09f8ac00425ff55177581258e4703f801942c" exitCode=6
Jul 17 22:19:27.662842 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:27.662444    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv" event={"ID":"d4d01405-f62c-4b81-bd39-3464260e1fd7","Type":"ContainerDied","Data":"28633b642daac3ee2bc1591a17c09f8ac00425ff55177581258e4703f801942c"}
Jul 17 22:19:27.662842 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:27.662516    2565 scope.go:117] "RemoveContainer" containerID="9c4735c5f8556b34fffc293e1a6c7f2fea653a6131a2ec65c0e4424657b8a4b2"
Jul 17 22:19:27.662842 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:27.662836    2565 scope.go:117] "RemoveContainer" containerID="28633b642daac3ee2bc1591a17c09f8ac00425ff55177581258e4703f801942c"
Jul 17 22:19:27.663125 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:19:27.663097    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv_odh-ai-gateway-infra(d4d01405-f62c-4b81-bd39-3464260e1fd7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv" podUID="d4d01405-f62c-4b81-bd39-3464260e1fd7"
Jul 17 22:19:27.664188 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:27.664173    2565 scope.go:117] "RemoveContainer" containerID="cd2e781d1c6b46f2f4b749d8349f18f66ac65e5d7dd68ed6a078976a7d06ab5e"
Jul 17 22:19:27.664354 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:19:27.664337    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh_odh-ai-gateway-infra(71894af1-cf6f-4960-85f7-5a8fbc942aae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh" podUID="71894af1-cf6f-4960-85f7-5a8fbc942aae"
Jul 17 22:19:27.691120 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:27.691084    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv"]
Jul 17 22:19:28.813362 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:28.813337    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv"
Jul 17 22:19:28.934126 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:28.933997    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d01405-f62c-4b81-bd39-3464260e1fd7-maas-api-service-ca\") pod \"d4d01405-f62c-4b81-bd39-3464260e1fd7\" (UID: \"d4d01405-f62c-4b81-bd39-3464260e1fd7\") "
Jul 17 22:19:28.934426 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:28.934401    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d01405-f62c-4b81-bd39-3464260e1fd7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4d01405-f62c-4b81-bd39-3464260e1fd7" (UID: "d4d01405-f62c-4b81-bd39-3464260e1fd7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:19:29.035663 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:29.035619    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d01405-f62c-4b81-bd39-3464260e1fd7-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:19:29.676328 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:29.676293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv" event={"ID":"d4d01405-f62c-4b81-bd39-3464260e1fd7","Type":"ContainerDied","Data":"4ba42846132a42f188fff12da082432fb0fd9369419552d8e6afaf8b23beec55"}
Jul 17 22:19:29.676328 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:29.676323    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv"
Jul 17 22:19:29.676597 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:29.676365    2565 scope.go:117] "RemoveContainer" containerID="28633b642daac3ee2bc1591a17c09f8ac00425ff55177581258e4703f801942c"
Jul 17 22:19:29.691902 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:29.691530    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv"]
Jul 17 22:19:29.693485 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:29.693461    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqtrv"]
Jul 17 22:19:31.182400 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:31.182367    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4d01405-f62c-4b81-bd39-3464260e1fd7" path="/var/lib/kubelet/pods/d4d01405-f62c-4b81-bd39-3464260e1fd7/volumes"
Jul 17 22:19:35.783338 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:35.783302    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj"]
Jul 17 22:19:35.783739 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:35.783697    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4d01405-f62c-4b81-bd39-3464260e1fd7" containerName="revoke-keys"
Jul 17 22:19:35.783739 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:35.783709    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4d01405-f62c-4b81-bd39-3464260e1fd7" containerName="revoke-keys"
Jul 17 22:19:35.783739 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:35.783719    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4d01405-f62c-4b81-bd39-3464260e1fd7" containerName="revoke-keys"
Jul 17 22:19:35.783739 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:35.783724    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4d01405-f62c-4b81-bd39-3464260e1fd7" containerName="revoke-keys"
Jul 17 22:19:35.783739 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:35.783737    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4d01405-f62c-4b81-bd39-3464260e1fd7" containerName="revoke-keys"
Jul 17 22:19:35.783894 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:35.783748    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4d01405-f62c-4b81-bd39-3464260e1fd7" containerName="revoke-keys"
Jul 17 22:19:35.783894 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:35.783805    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4d01405-f62c-4b81-bd39-3464260e1fd7" containerName="revoke-keys"
Jul 17 22:19:35.783894 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:35.783813    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4d01405-f62c-4b81-bd39-3464260e1fd7" containerName="revoke-keys"
Jul 17 22:19:35.783894 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:35.783820    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4d01405-f62c-4b81-bd39-3464260e1fd7" containerName="revoke-keys"
Jul 17 22:19:35.788268 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:35.788252    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj"
Jul 17 22:19:35.793941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:35.793914    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj"]
Jul 17 22:19:35.896928 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:35.896887    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj\" (UID: \"2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj"
Jul 17 22:19:35.998309 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:35.998267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj\" (UID: \"2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj"
Jul 17 22:19:35.998887 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:35.998869    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj\" (UID: \"2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj"
Jul 17 22:19:36.100956 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:36.100865    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj"
Jul 17 22:19:36.258248 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:36.258220    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj"]
Jul 17 22:19:36.261026 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:19:36.261000    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb4b1fb_c417_41d8_9aac_f88fcc02e4c2.slice/crio-98d0bfdfb3888404b79cf2a2e696d2db2be334bba5b6d2dc59da6bd2f6aff257 WatchSource:0}: Error finding container 98d0bfdfb3888404b79cf2a2e696d2db2be334bba5b6d2dc59da6bd2f6aff257: Status 404 returned error can't find the container with id 98d0bfdfb3888404b79cf2a2e696d2db2be334bba5b6d2dc59da6bd2f6aff257
Jul 17 22:19:36.710208 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:36.710105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj" event={"ID":"2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2","Type":"ContainerStarted","Data":"4fe74d87d6688c8230a391268a12b8de604b6396c3d117ceb6321ef170f67a2a"}
Jul 17 22:19:36.710208 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:36.710165    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj" event={"ID":"2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2","Type":"ContainerStarted","Data":"98d0bfdfb3888404b79cf2a2e696d2db2be334bba5b6d2dc59da6bd2f6aff257"}
Jul 17 22:19:36.726954 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:36.726900    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj" podStartSLOduration=1.726881138 podStartE2EDuration="1.726881138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:19:36.724643216 +0000 UTC m=+2040.113659866" watchObservedRunningTime="2026-07-17 22:19:36.726881138 +0000 UTC m=+2040.115897788"
Jul 17 22:19:37.716612 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:37.716506    2565 generic.go:356] "Generic (PLEG): container finished" podID="2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2" containerID="4fe74d87d6688c8230a391268a12b8de604b6396c3d117ceb6321ef170f67a2a" exitCode=6
Jul 17 22:19:37.716612 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:37.716595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj" event={"ID":"2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2","Type":"ContainerDied","Data":"4fe74d87d6688c8230a391268a12b8de604b6396c3d117ceb6321ef170f67a2a"}
Jul 17 22:19:37.717123 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:37.716742    2565 scope.go:117] "RemoveContainer" containerID="4fe74d87d6688c8230a391268a12b8de604b6396c3d117ceb6321ef170f67a2a"
Jul 17 22:19:38.722926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:38.722839    2565 generic.go:356] "Generic (PLEG): container finished" podID="2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2" containerID="edec71182b19e670ebd60dba8058692d5018a32e3aad2615cbda8dea955d07d6" exitCode=6
Jul 17 22:19:38.723378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:38.722926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj" event={"ID":"2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2","Type":"ContainerDied","Data":"edec71182b19e670ebd60dba8058692d5018a32e3aad2615cbda8dea955d07d6"}
Jul 17 22:19:38.723378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:38.722989    2565 scope.go:117] "RemoveContainer" containerID="4fe74d87d6688c8230a391268a12b8de604b6396c3d117ceb6321ef170f67a2a"
Jul 17 22:19:38.723378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:38.723239    2565 scope.go:117] "RemoveContainer" containerID="edec71182b19e670ebd60dba8058692d5018a32e3aad2615cbda8dea955d07d6"
Jul 17 22:19:38.723520 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:19:38.723434    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj_odh-ai-gateway-infra(2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj" podUID="2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2"
Jul 17 22:19:39.729521 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:39.729489    2565 scope.go:117] "RemoveContainer" containerID="edec71182b19e670ebd60dba8058692d5018a32e3aad2615cbda8dea955d07d6"
Jul 17 22:19:39.729965 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:19:39.729770    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj_odh-ai-gateway-infra(2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj" podUID="2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2"
Jul 17 22:19:42.177566 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:42.177536    2565 scope.go:117] "RemoveContainer" containerID="cd2e781d1c6b46f2f4b749d8349f18f66ac65e5d7dd68ed6a078976a7d06ab5e"
Jul 17 22:19:42.744604 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:42.744508    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh" event={"ID":"71894af1-cf6f-4960-85f7-5a8fbc942aae","Type":"ContainerStarted","Data":"9a3befddec45a08499cd5c65057d47b2446e3ef00a57cc3de854672a2730969d"}
Jul 17 22:19:42.760603 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:42.760552    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh" podStartSLOduration=18.760534544 podStartE2EDuration="18.760534544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:19:42.759746339 +0000 UTC m=+2046.148762991" watchObservedRunningTime="2026-07-17 22:19:42.760534544 +0000 UTC m=+2046.149551194"
Jul 17 22:19:43.751474 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:43.751437    2565 generic.go:356] "Generic (PLEG): container finished" podID="71894af1-cf6f-4960-85f7-5a8fbc942aae" containerID="9a3befddec45a08499cd5c65057d47b2446e3ef00a57cc3de854672a2730969d" exitCode=7
Jul 17 22:19:43.751890 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:43.751510    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh" event={"ID":"71894af1-cf6f-4960-85f7-5a8fbc942aae","Type":"ContainerDied","Data":"9a3befddec45a08499cd5c65057d47b2446e3ef00a57cc3de854672a2730969d"}
Jul 17 22:19:43.751890 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:43.751573    2565 scope.go:117] "RemoveContainer" containerID="cd2e781d1c6b46f2f4b749d8349f18f66ac65e5d7dd68ed6a078976a7d06ab5e"
Jul 17 22:19:43.751890 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:43.751871    2565 scope.go:117] "RemoveContainer" containerID="9a3befddec45a08499cd5c65057d47b2446e3ef00a57cc3de854672a2730969d"
Jul 17 22:19:43.752129 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:19:43.752110    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh_odh-ai-gateway-infra(71894af1-cf6f-4960-85f7-5a8fbc942aae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh" podUID="71894af1-cf6f-4960-85f7-5a8fbc942aae"
Jul 17 22:19:43.777250 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:43.777212    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh"]
Jul 17 22:19:44.906842 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:44.906819    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh"
Jul 17 22:19:44.983613 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:44.983575    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71894af1-cf6f-4960-85f7-5a8fbc942aae-maas-api-service-ca\") pod \"71894af1-cf6f-4960-85f7-5a8fbc942aae\" (UID: \"71894af1-cf6f-4960-85f7-5a8fbc942aae\") "
Jul 17 22:19:44.983949 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:44.983927    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71894af1-cf6f-4960-85f7-5a8fbc942aae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "71894af1-cf6f-4960-85f7-5a8fbc942aae" (UID: "71894af1-cf6f-4960-85f7-5a8fbc942aae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:19:45.085211 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:45.085118    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71894af1-cf6f-4960-85f7-5a8fbc942aae-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:19:45.763583 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:45.763551    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh"
Jul 17 22:19:45.763768 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:45.763548    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh" event={"ID":"71894af1-cf6f-4960-85f7-5a8fbc942aae","Type":"ContainerDied","Data":"9b783caaf46a796389b3bd3646f8c8ae333134a2d86e50f9f33d81cdf80438f5"}
Jul 17 22:19:45.763768 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:45.763681    2565 scope.go:117] "RemoveContainer" containerID="9a3befddec45a08499cd5c65057d47b2446e3ef00a57cc3de854672a2730969d"
Jul 17 22:19:45.780648 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:45.780618    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh"]
Jul 17 22:19:45.786220 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:45.786191    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-6q5kh"]
Jul 17 22:19:47.182189 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:47.182155    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71894af1-cf6f-4960-85f7-5a8fbc942aae" path="/var/lib/kubelet/pods/71894af1-cf6f-4960-85f7-5a8fbc942aae/volumes"
Jul 17 22:19:51.183255 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:51.183226    2565 scope.go:117] "RemoveContainer" containerID="edec71182b19e670ebd60dba8058692d5018a32e3aad2615cbda8dea955d07d6"
Jul 17 22:19:51.794450 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:51.794417    2565 generic.go:356] "Generic (PLEG): container finished" podID="2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2" containerID="b676d4f3e68fa8e1037c2a34e3e68622493ef95d621b0a33f3521563b3639890" exitCode=6
Jul 17 22:19:51.794640 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:51.794492    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj" event={"ID":"2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2","Type":"ContainerDied","Data":"b676d4f3e68fa8e1037c2a34e3e68622493ef95d621b0a33f3521563b3639890"}
Jul 17 22:19:51.794640 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:51.794559    2565 scope.go:117] "RemoveContainer" containerID="edec71182b19e670ebd60dba8058692d5018a32e3aad2615cbda8dea955d07d6"
Jul 17 22:19:51.794916 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:51.794901    2565 scope.go:117] "RemoveContainer" containerID="b676d4f3e68fa8e1037c2a34e3e68622493ef95d621b0a33f3521563b3639890"
Jul 17 22:19:51.795179 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:19:51.795159    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj_odh-ai-gateway-infra(2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj" podUID="2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2"
Jul 17 22:19:52.819840 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:52.819805    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj"]
Jul 17 22:19:52.956938 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:52.956911    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj"
Jul 17 22:19:53.061124 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:53.061061    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2-maas-api-service-ca\") pod \"2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2\" (UID: \"2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2\") "
Jul 17 22:19:53.061460 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:53.061428    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2" (UID: "2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:19:53.162686 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:53.162584    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:19:53.809471 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:53.809435    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj" event={"ID":"2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2","Type":"ContainerDied","Data":"98d0bfdfb3888404b79cf2a2e696d2db2be334bba5b6d2dc59da6bd2f6aff257"}
Jul 17 22:19:53.809670 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:53.809500    2565 scope.go:117] "RemoveContainer" containerID="b676d4f3e68fa8e1037c2a34e3e68622493ef95d621b0a33f3521563b3639890"
Jul 17 22:19:53.809670 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:53.809452    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj"
Jul 17 22:19:53.825898 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:53.825860    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj"]
Jul 17 22:19:53.831627 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:53.831599    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjtwj"]
Jul 17 22:19:55.183701 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:19:55.183663    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2" path="/var/lib/kubelet/pods/2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2/volumes"
Jul 17 22:20:07.628458 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.628372    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6"]
Jul 17 22:20:07.628823 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.628752    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71894af1-cf6f-4960-85f7-5a8fbc942aae" containerName="revoke-keys"
Jul 17 22:20:07.628823 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.628763    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71894af1-cf6f-4960-85f7-5a8fbc942aae" containerName="revoke-keys"
Jul 17 22:20:07.628823 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.628772    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2" containerName="revoke-keys"
Jul 17 22:20:07.628823 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.628778    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2" containerName="revoke-keys"
Jul 17 22:20:07.628823 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.628790    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2" containerName="revoke-keys"
Jul 17 22:20:07.628823 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.628796    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2" containerName="revoke-keys"
Jul 17 22:20:07.628823 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.628809    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71894af1-cf6f-4960-85f7-5a8fbc942aae" containerName="revoke-keys"
Jul 17 22:20:07.628823 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.628814    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71894af1-cf6f-4960-85f7-5a8fbc942aae" containerName="revoke-keys"
Jul 17 22:20:07.629098 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.628863    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71894af1-cf6f-4960-85f7-5a8fbc942aae" containerName="revoke-keys"
Jul 17 22:20:07.629098 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.628870    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71894af1-cf6f-4960-85f7-5a8fbc942aae" containerName="revoke-keys"
Jul 17 22:20:07.629098 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.628875    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2" containerName="revoke-keys"
Jul 17 22:20:07.629098 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.628881    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71894af1-cf6f-4960-85f7-5a8fbc942aae" containerName="revoke-keys"
Jul 17 22:20:07.629098 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.628888    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2" containerName="revoke-keys"
Jul 17 22:20:07.633113 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.633093    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6"
Jul 17 22:20:07.635594 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.635565    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:20:07.635594 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.635566    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:20:07.639488 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.639460    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6"]
Jul 17 22:20:07.700774 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.700724    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c89c821-fffd-43c7-b080-559c44049739-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6\" (UID: \"2c89c821-fffd-43c7-b080-559c44049739\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6"
Jul 17 22:20:07.802183 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.802138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c89c821-fffd-43c7-b080-559c44049739-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6\" (UID: \"2c89c821-fffd-43c7-b080-559c44049739\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6"
Jul 17 22:20:07.802767 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.802748    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c89c821-fffd-43c7-b080-559c44049739-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6\" (UID: \"2c89c821-fffd-43c7-b080-559c44049739\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6"
Jul 17 22:20:07.946861 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:07.946773    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6"
Jul 17 22:20:08.085577 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:08.085545    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6"]
Jul 17 22:20:08.088035 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:20:08.088005    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c89c821_fffd_43c7_b080_559c44049739.slice/crio-8a3966da562ff98f392a854e24634016ddb3fc89a4d5ad9c1beb63ba14b77a3b WatchSource:0}: Error finding container 8a3966da562ff98f392a854e24634016ddb3fc89a4d5ad9c1beb63ba14b77a3b: Status 404 returned error can't find the container with id 8a3966da562ff98f392a854e24634016ddb3fc89a4d5ad9c1beb63ba14b77a3b
Jul 17 22:20:08.888953 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:08.888914    2565 generic.go:356] "Generic (PLEG): container finished" podID="2c89c821-fffd-43c7-b080-559c44049739" containerID="7398871c02f3383581c19c27659e2a46dee77d1c95335fcbcb8c89085f3198a1" exitCode=6
Jul 17 22:20:08.889372 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:08.888970    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6" event={"ID":"2c89c821-fffd-43c7-b080-559c44049739","Type":"ContainerDied","Data":"7398871c02f3383581c19c27659e2a46dee77d1c95335fcbcb8c89085f3198a1"}
Jul 17 22:20:08.889372 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:08.889033    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6" event={"ID":"2c89c821-fffd-43c7-b080-559c44049739","Type":"ContainerStarted","Data":"8a3966da562ff98f392a854e24634016ddb3fc89a4d5ad9c1beb63ba14b77a3b"}
Jul 17 22:20:08.889372 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:08.889245    2565 scope.go:117] "RemoveContainer" containerID="7398871c02f3383581c19c27659e2a46dee77d1c95335fcbcb8c89085f3198a1"
Jul 17 22:20:09.895729 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:09.895698    2565 generic.go:356] "Generic (PLEG): container finished" podID="2c89c821-fffd-43c7-b080-559c44049739" containerID="a2d5672a4b729dfbbe6681b969a145b7b159b9714e02cd2e3d6f3e8bb06cafa3" exitCode=6
Jul 17 22:20:09.896157 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:09.895758    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6" event={"ID":"2c89c821-fffd-43c7-b080-559c44049739","Type":"ContainerDied","Data":"a2d5672a4b729dfbbe6681b969a145b7b159b9714e02cd2e3d6f3e8bb06cafa3"}
Jul 17 22:20:09.896157 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:09.895807    2565 scope.go:117] "RemoveContainer" containerID="7398871c02f3383581c19c27659e2a46dee77d1c95335fcbcb8c89085f3198a1"
Jul 17 22:20:09.896157 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:09.895975    2565 scope.go:117] "RemoveContainer" containerID="a2d5672a4b729dfbbe6681b969a145b7b159b9714e02cd2e3d6f3e8bb06cafa3"
Jul 17 22:20:09.896305 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:20:09.896241    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6_odh-ai-gateway-infra(2c89c821-fffd-43c7-b080-559c44049739)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6" podUID="2c89c821-fffd-43c7-b080-559c44049739"
Jul 17 22:20:10.902211 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:10.902182    2565 scope.go:117] "RemoveContainer" containerID="a2d5672a4b729dfbbe6681b969a145b7b159b9714e02cd2e3d6f3e8bb06cafa3"
Jul 17 22:20:10.902623 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:20:10.902401    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6_odh-ai-gateway-infra(2c89c821-fffd-43c7-b080-559c44049739)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6" podUID="2c89c821-fffd-43c7-b080-559c44049739"
Jul 17 22:20:24.300920 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:24.300882    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz"]
Jul 17 22:20:24.301379 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:24.301262    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71894af1-cf6f-4960-85f7-5a8fbc942aae" containerName="revoke-keys"
Jul 17 22:20:24.301379 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:24.301274    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71894af1-cf6f-4960-85f7-5a8fbc942aae" containerName="revoke-keys"
Jul 17 22:20:24.301379 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:24.301283    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2" containerName="revoke-keys"
Jul 17 22:20:24.301379 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:24.301288    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2" containerName="revoke-keys"
Jul 17 22:20:24.301379 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:24.301353    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cb4b1fb-c417-41d8-9aac-f88fcc02e4c2" containerName="revoke-keys"
Jul 17 22:20:24.306059 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:24.306033    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz"
Jul 17 22:20:24.311776 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:24.311746    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz"]
Jul 17 22:20:24.455497 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:24.455455    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9b921c-947a-4a34-ad27-6d7861034cc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz\" (UID: \"4e9b921c-947a-4a34-ad27-6d7861034cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz"
Jul 17 22:20:24.556830 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:24.556731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9b921c-947a-4a34-ad27-6d7861034cc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz\" (UID: \"4e9b921c-947a-4a34-ad27-6d7861034cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz"
Jul 17 22:20:24.557392 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:24.557372    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9b921c-947a-4a34-ad27-6d7861034cc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz\" (UID: \"4e9b921c-947a-4a34-ad27-6d7861034cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz"
Jul 17 22:20:24.618257 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:24.618219    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz"
Jul 17 22:20:24.755261 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:24.755226    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz"]
Jul 17 22:20:24.758690 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:20:24.758657    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9b921c_947a_4a34_ad27_6d7861034cc0.slice/crio-906b8c2ef36c8738e104e4f35333bfb85629716ff08af0e6b71286257daaadbf WatchSource:0}: Error finding container 906b8c2ef36c8738e104e4f35333bfb85629716ff08af0e6b71286257daaadbf: Status 404 returned error can't find the container with id 906b8c2ef36c8738e104e4f35333bfb85629716ff08af0e6b71286257daaadbf
Jul 17 22:20:24.968554 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:24.968527    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz" event={"ID":"4e9b921c-947a-4a34-ad27-6d7861034cc0","Type":"ContainerStarted","Data":"906b8c2ef36c8738e104e4f35333bfb85629716ff08af0e6b71286257daaadbf"}
Jul 17 22:20:25.176772 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:25.176672    2565 scope.go:117] "RemoveContainer" containerID="a2d5672a4b729dfbbe6681b969a145b7b159b9714e02cd2e3d6f3e8bb06cafa3"
Jul 17 22:20:25.822595 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:25.822556    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b"]
Jul 17 22:20:25.826097 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:25.826053    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b"
Jul 17 22:20:25.832057 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:25.831994    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b"]
Jul 17 22:20:25.968882 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:25.968822    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2635e5-252f-4386-a1ad-7002ba421d2b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-shg8b\" (UID: \"7e2635e5-252f-4386-a1ad-7002ba421d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b"
Jul 17 22:20:25.974890 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:25.974860    2565 generic.go:356] "Generic (PLEG): container finished" podID="2c89c821-fffd-43c7-b080-559c44049739" containerID="4e292d38629a4861edd041cccdf5145080a7b2902a5478f0f2e92024b7d2b391" exitCode=6
Jul 17 22:20:25.975110 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:25.974939    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6" event={"ID":"2c89c821-fffd-43c7-b080-559c44049739","Type":"ContainerDied","Data":"4e292d38629a4861edd041cccdf5145080a7b2902a5478f0f2e92024b7d2b391"}
Jul 17 22:20:25.975110 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:25.975014    2565 scope.go:117] "RemoveContainer" containerID="a2d5672a4b729dfbbe6681b969a145b7b159b9714e02cd2e3d6f3e8bb06cafa3"
Jul 17 22:20:25.975343 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:25.975315    2565 scope.go:117] "RemoveContainer" containerID="4e292d38629a4861edd041cccdf5145080a7b2902a5478f0f2e92024b7d2b391"
Jul 17 22:20:25.975597 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:20:25.975566    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6_odh-ai-gateway-infra(2c89c821-fffd-43c7-b080-559c44049739)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6" podUID="2c89c821-fffd-43c7-b080-559c44049739"
Jul 17 22:20:25.985315 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:25.985285    2565 generic.go:356] "Generic (PLEG): container finished" podID="4e9b921c-947a-4a34-ad27-6d7861034cc0" containerID="ca39fb4f9e9ba25e7631b6a94ad63a4254fe8407fe08c9d3aca9743b9c5bd94c" exitCode=7
Jul 17 22:20:25.985439 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:25.985361    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz" event={"ID":"4e9b921c-947a-4a34-ad27-6d7861034cc0","Type":"ContainerDied","Data":"ca39fb4f9e9ba25e7631b6a94ad63a4254fe8407fe08c9d3aca9743b9c5bd94c"}
Jul 17 22:20:25.985557 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:25.985543    2565 scope.go:117] "RemoveContainer" containerID="ca39fb4f9e9ba25e7631b6a94ad63a4254fe8407fe08c9d3aca9743b9c5bd94c"
Jul 17 22:20:26.070417 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:26.070378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2635e5-252f-4386-a1ad-7002ba421d2b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-shg8b\" (UID: \"7e2635e5-252f-4386-a1ad-7002ba421d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b"
Jul 17 22:20:26.071098 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:26.071052    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2635e5-252f-4386-a1ad-7002ba421d2b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-shg8b\" (UID: \"7e2635e5-252f-4386-a1ad-7002ba421d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b"
Jul 17 22:20:26.137678 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:26.137575    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b"
Jul 17 22:20:26.279452 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:26.279427    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b"]
Jul 17 22:20:26.281726 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:20:26.281693    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2635e5_252f_4386_a1ad_7002ba421d2b.slice/crio-8cc6d618765402266c5dfbad076cc1fe935f1a96425365300459c7d3a0b34c8f WatchSource:0}: Error finding container 8cc6d618765402266c5dfbad076cc1fe935f1a96425365300459c7d3a0b34c8f: Status 404 returned error can't find the container with id 8cc6d618765402266c5dfbad076cc1fe935f1a96425365300459c7d3a0b34c8f
Jul 17 22:20:26.993220 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:26.993184    2565 generic.go:356] "Generic (PLEG): container finished" podID="7e2635e5-252f-4386-a1ad-7002ba421d2b" containerID="3f756db3066f7f4cac110cfe45b95ef1ba1d5e1309db84dc46929a68f7fb07b2" exitCode=6
Jul 17 22:20:26.993690 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:26.993269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b" event={"ID":"7e2635e5-252f-4386-a1ad-7002ba421d2b","Type":"ContainerDied","Data":"3f756db3066f7f4cac110cfe45b95ef1ba1d5e1309db84dc46929a68f7fb07b2"}
Jul 17 22:20:26.993690 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:26.993328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b" event={"ID":"7e2635e5-252f-4386-a1ad-7002ba421d2b","Type":"ContainerStarted","Data":"8cc6d618765402266c5dfbad076cc1fe935f1a96425365300459c7d3a0b34c8f"}
Jul 17 22:20:26.993690 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:26.993463    2565 scope.go:117] "RemoveContainer" containerID="3f756db3066f7f4cac110cfe45b95ef1ba1d5e1309db84dc46929a68f7fb07b2"
Jul 17 22:20:26.995232 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:26.995210    2565 generic.go:356] "Generic (PLEG): container finished" podID="4e9b921c-947a-4a34-ad27-6d7861034cc0" containerID="a833387e4278c6a54e0d3265b381a5a1089c0c5a93b5f177e7155e1c2ba99996" exitCode=7
Jul 17 22:20:26.995315 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:26.995289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz" event={"ID":"4e9b921c-947a-4a34-ad27-6d7861034cc0","Type":"ContainerDied","Data":"a833387e4278c6a54e0d3265b381a5a1089c0c5a93b5f177e7155e1c2ba99996"}
Jul 17 22:20:26.995355 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:26.995336    2565 scope.go:117] "RemoveContainer" containerID="ca39fb4f9e9ba25e7631b6a94ad63a4254fe8407fe08c9d3aca9743b9c5bd94c"
Jul 17 22:20:26.995479 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:26.995456    2565 scope.go:117] "RemoveContainer" containerID="a833387e4278c6a54e0d3265b381a5a1089c0c5a93b5f177e7155e1c2ba99996"
Jul 17 22:20:26.995845 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:20:26.995723    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz_odh-ai-gateway-infra(4e9b921c-947a-4a34-ad27-6d7861034cc0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz" podUID="4e9b921c-947a-4a34-ad27-6d7861034cc0"
Jul 17 22:20:27.002796 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:27.002774    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6"]
Jul 17 22:20:27.139367 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:27.139343    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6"
Jul 17 22:20:27.281922 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:27.281892    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c89c821-fffd-43c7-b080-559c44049739-maas-api-service-ca\") pod \"2c89c821-fffd-43c7-b080-559c44049739\" (UID: \"2c89c821-fffd-43c7-b080-559c44049739\") "
Jul 17 22:20:27.282343 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:27.282322    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c89c821-fffd-43c7-b080-559c44049739-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c89c821-fffd-43c7-b080-559c44049739" (UID: "2c89c821-fffd-43c7-b080-559c44049739"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:20:27.383765 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:27.383674    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c89c821-fffd-43c7-b080-559c44049739-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:20:28.000855 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:28.000830    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6"
Jul 17 22:20:28.001334 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:28.000826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6" event={"ID":"2c89c821-fffd-43c7-b080-559c44049739","Type":"ContainerDied","Data":"8a3966da562ff98f392a854e24634016ddb3fc89a4d5ad9c1beb63ba14b77a3b"}
Jul 17 22:20:28.001334 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:28.000974    2565 scope.go:117] "RemoveContainer" containerID="4e292d38629a4861edd041cccdf5145080a7b2902a5478f0f2e92024b7d2b391"
Jul 17 22:20:28.002814 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:28.002783    2565 generic.go:356] "Generic (PLEG): container finished" podID="7e2635e5-252f-4386-a1ad-7002ba421d2b" containerID="4baa23da8cf72b2f93cd8cd0abcc4f0eda633087264734e3964dc2cde9001c89" exitCode=6
Jul 17 22:20:28.002950 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:28.002860    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b" event={"ID":"7e2635e5-252f-4386-a1ad-7002ba421d2b","Type":"ContainerDied","Data":"4baa23da8cf72b2f93cd8cd0abcc4f0eda633087264734e3964dc2cde9001c89"}
Jul 17 22:20:28.003145 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:28.003125    2565 scope.go:117] "RemoveContainer" containerID="4baa23da8cf72b2f93cd8cd0abcc4f0eda633087264734e3964dc2cde9001c89"
Jul 17 22:20:28.003409 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:20:28.003388    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-shg8b_odh-ai-gateway-infra(7e2635e5-252f-4386-a1ad-7002ba421d2b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b" podUID="7e2635e5-252f-4386-a1ad-7002ba421d2b"
Jul 17 22:20:28.004996 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:28.004980    2565 scope.go:117] "RemoveContainer" containerID="a833387e4278c6a54e0d3265b381a5a1089c0c5a93b5f177e7155e1c2ba99996"
Jul 17 22:20:28.005194 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:20:28.005175    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz_odh-ai-gateway-infra(4e9b921c-947a-4a34-ad27-6d7861034cc0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz" podUID="4e9b921c-947a-4a34-ad27-6d7861034cc0"
Jul 17 22:20:28.009907 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:28.009890    2565 scope.go:117] "RemoveContainer" containerID="3f756db3066f7f4cac110cfe45b95ef1ba1d5e1309db84dc46929a68f7fb07b2"
Jul 17 22:20:28.049028 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:28.048995    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6"]
Jul 17 22:20:28.052634 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:28.052604    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2x4m6"]
Jul 17 22:20:29.013689 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:29.013661    2565 scope.go:117] "RemoveContainer" containerID="4baa23da8cf72b2f93cd8cd0abcc4f0eda633087264734e3964dc2cde9001c89"
Jul 17 22:20:29.014089 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:20:29.013872    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-shg8b_odh-ai-gateway-infra(7e2635e5-252f-4386-a1ad-7002ba421d2b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b" podUID="7e2635e5-252f-4386-a1ad-7002ba421d2b"
Jul 17 22:20:29.182720 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:29.182687    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c89c821-fffd-43c7-b080-559c44049739" path="/var/lib/kubelet/pods/2c89c821-fffd-43c7-b080-559c44049739/volumes"
Jul 17 22:20:29.580667 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:29.580630    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:20:29.664500 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:29.664448    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:20:37.240722 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:37.240689    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:20:37.248219 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:37.248198    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:20:38.176924 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:38.176892    2565 scope.go:117] "RemoveContainer" containerID="a833387e4278c6a54e0d3265b381a5a1089c0c5a93b5f177e7155e1c2ba99996"
Jul 17 22:20:39.068135 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:39.068024    2565 generic.go:356] "Generic (PLEG): container finished" podID="4e9b921c-947a-4a34-ad27-6d7861034cc0" containerID="6f57c94ab024483076abb899278e8e5d1c29dbeff42c706519593aa23c17fd61" exitCode=7
Jul 17 22:20:39.068135 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:39.068101    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz" event={"ID":"4e9b921c-947a-4a34-ad27-6d7861034cc0","Type":"ContainerDied","Data":"6f57c94ab024483076abb899278e8e5d1c29dbeff42c706519593aa23c17fd61"}
Jul 17 22:20:39.068642 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:39.068166    2565 scope.go:117] "RemoveContainer" containerID="a833387e4278c6a54e0d3265b381a5a1089c0c5a93b5f177e7155e1c2ba99996"
Jul 17 22:20:39.068642 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:39.068491    2565 scope.go:117] "RemoveContainer" containerID="6f57c94ab024483076abb899278e8e5d1c29dbeff42c706519593aa23c17fd61"
Jul 17 22:20:39.068790 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:20:39.068767    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz_odh-ai-gateway-infra(4e9b921c-947a-4a34-ad27-6d7861034cc0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz" podUID="4e9b921c-947a-4a34-ad27-6d7861034cc0"
Jul 17 22:20:40.095730 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:40.095697    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz"]
Jul 17 22:20:40.237990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:40.237962    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz"
Jul 17 22:20:40.308783 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:40.308744    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9b921c-947a-4a34-ad27-6d7861034cc0-maas-api-service-ca\") pod \"4e9b921c-947a-4a34-ad27-6d7861034cc0\" (UID: \"4e9b921c-947a-4a34-ad27-6d7861034cc0\") "
Jul 17 22:20:40.309147 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:40.309126    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9b921c-947a-4a34-ad27-6d7861034cc0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e9b921c-947a-4a34-ad27-6d7861034cc0" (UID: "4e9b921c-947a-4a34-ad27-6d7861034cc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:20:40.410482 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:40.410387    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9b921c-947a-4a34-ad27-6d7861034cc0-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:20:41.082556 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:41.082525    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz"
Jul 17 22:20:41.082763 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:41.082572    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz" event={"ID":"4e9b921c-947a-4a34-ad27-6d7861034cc0","Type":"ContainerDied","Data":"906b8c2ef36c8738e104e4f35333bfb85629716ff08af0e6b71286257daaadbf"}
Jul 17 22:20:41.082763 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:41.082662    2565 scope.go:117] "RemoveContainer" containerID="6f57c94ab024483076abb899278e8e5d1c29dbeff42c706519593aa23c17fd61"
Jul 17 22:20:41.102920 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:41.102886    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz"]
Jul 17 22:20:41.109389 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:41.109360    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-hn4pz"]
Jul 17 22:20:41.182802 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:41.182768    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e9b921c-947a-4a34-ad27-6d7861034cc0" path="/var/lib/kubelet/pods/4e9b921c-947a-4a34-ad27-6d7861034cc0/volumes"
Jul 17 22:20:43.177143 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:43.177107    2565 scope.go:117] "RemoveContainer" containerID="4baa23da8cf72b2f93cd8cd0abcc4f0eda633087264734e3964dc2cde9001c89"
Jul 17 22:20:44.099688 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:44.099655    2565 generic.go:356] "Generic (PLEG): container finished" podID="7e2635e5-252f-4386-a1ad-7002ba421d2b" containerID="1b39d23def8bae4e05909fe222eb1b50ce906a4c02150b5e79c6d46238a2e64b" exitCode=6
Jul 17 22:20:44.099868 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:44.099733    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b" event={"ID":"7e2635e5-252f-4386-a1ad-7002ba421d2b","Type":"ContainerDied","Data":"1b39d23def8bae4e05909fe222eb1b50ce906a4c02150b5e79c6d46238a2e64b"}
Jul 17 22:20:44.099868 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:44.099794    2565 scope.go:117] "RemoveContainer" containerID="4baa23da8cf72b2f93cd8cd0abcc4f0eda633087264734e3964dc2cde9001c89"
Jul 17 22:20:44.100153 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:44.100138    2565 scope.go:117] "RemoveContainer" containerID="1b39d23def8bae4e05909fe222eb1b50ce906a4c02150b5e79c6d46238a2e64b"
Jul 17 22:20:44.100385 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:20:44.100368    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-shg8b_odh-ai-gateway-infra(7e2635e5-252f-4386-a1ad-7002ba421d2b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b" podUID="7e2635e5-252f-4386-a1ad-7002ba421d2b"
Jul 17 22:20:44.575835 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:44.575802    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:20:44.677207 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:44.677155    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:20:45.128771 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:45.128737    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b"]
Jul 17 22:20:45.263989 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:45.263962    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b"
Jul 17 22:20:45.354178 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:45.354141    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2635e5-252f-4386-a1ad-7002ba421d2b-maas-api-service-ca\") pod \"7e2635e5-252f-4386-a1ad-7002ba421d2b\" (UID: \"7e2635e5-252f-4386-a1ad-7002ba421d2b\") "
Jul 17 22:20:45.354553 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:45.354522    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2635e5-252f-4386-a1ad-7002ba421d2b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e2635e5-252f-4386-a1ad-7002ba421d2b" (UID: "7e2635e5-252f-4386-a1ad-7002ba421d2b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:20:45.455988 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:45.455900    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2635e5-252f-4386-a1ad-7002ba421d2b-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:20:46.118936 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:46.118901    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b" event={"ID":"7e2635e5-252f-4386-a1ad-7002ba421d2b","Type":"ContainerDied","Data":"8cc6d618765402266c5dfbad076cc1fe935f1a96425365300459c7d3a0b34c8f"}
Jul 17 22:20:46.119394 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:46.118943    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b"
Jul 17 22:20:46.119394 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:46.118986    2565 scope.go:117] "RemoveContainer" containerID="1b39d23def8bae4e05909fe222eb1b50ce906a4c02150b5e79c6d46238a2e64b"
Jul 17 22:20:46.142696 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:46.142671    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b"]
Jul 17 22:20:46.146571 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:46.146549    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-shg8b"]
Jul 17 22:20:47.182982 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:20:47.182948    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e2635e5-252f-4386-a1ad-7002ba421d2b" path="/var/lib/kubelet/pods/7e2635e5-252f-4386-a1ad-7002ba421d2b/volumes"
Jul 17 22:21:07.673906 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.673870    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g"]
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674240    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e9b921c-947a-4a34-ad27-6d7861034cc0" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674252    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e9b921c-947a-4a34-ad27-6d7861034cc0" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674260    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e9b921c-947a-4a34-ad27-6d7861034cc0" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674266    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e9b921c-947a-4a34-ad27-6d7861034cc0" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674276    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c89c821-fffd-43c7-b080-559c44049739" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674282    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c89c821-fffd-43c7-b080-559c44049739" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674289    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c89c821-fffd-43c7-b080-559c44049739" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674294    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c89c821-fffd-43c7-b080-559c44049739" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674300    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e2635e5-252f-4386-a1ad-7002ba421d2b" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674307    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e2635e5-252f-4386-a1ad-7002ba421d2b" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674315    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c89c821-fffd-43c7-b080-559c44049739" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674320    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c89c821-fffd-43c7-b080-559c44049739" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674332    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e2635e5-252f-4386-a1ad-7002ba421d2b" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674337    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e2635e5-252f-4386-a1ad-7002ba421d2b" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674342    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e2635e5-252f-4386-a1ad-7002ba421d2b" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674347    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e2635e5-252f-4386-a1ad-7002ba421d2b" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674401    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e9b921c-947a-4a34-ad27-6d7861034cc0" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674408    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c89c821-fffd-43c7-b080-559c44049739" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674412    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c89c821-fffd-43c7-b080-559c44049739" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674418    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c89c821-fffd-43c7-b080-559c44049739" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674423    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e2635e5-252f-4386-a1ad-7002ba421d2b" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674428    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e2635e5-252f-4386-a1ad-7002ba421d2b" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674435    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e2635e5-252f-4386-a1ad-7002ba421d2b" containerName="revoke-keys"
Jul 17 22:21:07.674430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.674442    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e9b921c-947a-4a34-ad27-6d7861034cc0" containerName="revoke-keys"
Jul 17 22:21:07.678991 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.678970    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g"
Jul 17 22:21:07.682443 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.682417    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:21:07.682808 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.682461    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:21:07.684168 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.684137    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g"]
Jul 17 22:21:07.765880 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.765841    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a36b1ee2-c4b5-4a8f-8356-7dd5ded76791-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g\" (UID: \"a36b1ee2-c4b5-4a8f-8356-7dd5ded76791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g"
Jul 17 22:21:07.866996 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.866941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a36b1ee2-c4b5-4a8f-8356-7dd5ded76791-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g\" (UID: \"a36b1ee2-c4b5-4a8f-8356-7dd5ded76791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g"
Jul 17 22:21:07.867614 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.867592    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a36b1ee2-c4b5-4a8f-8356-7dd5ded76791-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g\" (UID: \"a36b1ee2-c4b5-4a8f-8356-7dd5ded76791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g"
Jul 17 22:21:07.991935 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:07.991838    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g"
Jul 17 22:21:08.130281 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:08.130176    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g"]
Jul 17 22:21:08.132702 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:21:08.132668    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36b1ee2_c4b5_4a8f_8356_7dd5ded76791.slice/crio-88448d8c5e59624e783591207f49b33b1f8e6193c38930cab0e830eca0687cfe WatchSource:0}: Error finding container 88448d8c5e59624e783591207f49b33b1f8e6193c38930cab0e830eca0687cfe: Status 404 returned error can't find the container with id 88448d8c5e59624e783591207f49b33b1f8e6193c38930cab0e830eca0687cfe
Jul 17 22:21:08.225209 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:08.225175    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g" event={"ID":"a36b1ee2-c4b5-4a8f-8356-7dd5ded76791","Type":"ContainerStarted","Data":"88448d8c5e59624e783591207f49b33b1f8e6193c38930cab0e830eca0687cfe"}
Jul 17 22:21:09.231751 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:09.231650    2565 generic.go:356] "Generic (PLEG): container finished" podID="a36b1ee2-c4b5-4a8f-8356-7dd5ded76791" containerID="76378ec7d69bf592d8fbf612a7fb68f52351a14fdd12fb46ca599de459b40170" exitCode=6
Jul 17 22:21:09.232150 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:09.231745    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g" event={"ID":"a36b1ee2-c4b5-4a8f-8356-7dd5ded76791","Type":"ContainerDied","Data":"76378ec7d69bf592d8fbf612a7fb68f52351a14fdd12fb46ca599de459b40170"}
Jul 17 22:21:09.232150 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:09.231924    2565 scope.go:117] "RemoveContainer" containerID="76378ec7d69bf592d8fbf612a7fb68f52351a14fdd12fb46ca599de459b40170"
Jul 17 22:21:10.239445 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:10.239352    2565 generic.go:356] "Generic (PLEG): container finished" podID="a36b1ee2-c4b5-4a8f-8356-7dd5ded76791" containerID="c2beaa20eb6a846fa237bf99ac3608098fb8a6dca47e9db8b958c9d57f833fa1" exitCode=6
Jul 17 22:21:10.239445 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:10.239389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g" event={"ID":"a36b1ee2-c4b5-4a8f-8356-7dd5ded76791","Type":"ContainerDied","Data":"c2beaa20eb6a846fa237bf99ac3608098fb8a6dca47e9db8b958c9d57f833fa1"}
Jul 17 22:21:10.239445 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:10.239436    2565 scope.go:117] "RemoveContainer" containerID="76378ec7d69bf592d8fbf612a7fb68f52351a14fdd12fb46ca599de459b40170"
Jul 17 22:21:10.239981 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:10.239713    2565 scope.go:117] "RemoveContainer" containerID="c2beaa20eb6a846fa237bf99ac3608098fb8a6dca47e9db8b958c9d57f833fa1"
Jul 17 22:21:10.239981 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:21:10.239940    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g_odh-ai-gateway-infra(a36b1ee2-c4b5-4a8f-8356-7dd5ded76791)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g" podUID="a36b1ee2-c4b5-4a8f-8356-7dd5ded76791"
Jul 17 22:21:11.245463 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:11.245435    2565 scope.go:117] "RemoveContainer" containerID="c2beaa20eb6a846fa237bf99ac3608098fb8a6dca47e9db8b958c9d57f833fa1"
Jul 17 22:21:11.245847 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:21:11.245619    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g_odh-ai-gateway-infra(a36b1ee2-c4b5-4a8f-8356-7dd5ded76791)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g" podUID="a36b1ee2-c4b5-4a8f-8356-7dd5ded76791"
Jul 17 22:21:14.343012 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:14.342969    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5"]
Jul 17 22:21:14.343418 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:14.343348    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e9b921c-947a-4a34-ad27-6d7861034cc0" containerName="revoke-keys"
Jul 17 22:21:14.343418 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:14.343359    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e9b921c-947a-4a34-ad27-6d7861034cc0" containerName="revoke-keys"
Jul 17 22:21:14.343489 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:14.343419    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e9b921c-947a-4a34-ad27-6d7861034cc0" containerName="revoke-keys"
Jul 17 22:21:14.347542 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:14.347524    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5"
Jul 17 22:21:14.353788 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:14.353632    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5"]
Jul 17 22:21:14.432985 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:14.432941    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da87ad45-6841-485c-afcc-597845c15442-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5\" (UID: \"da87ad45-6841-485c-afcc-597845c15442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5"
Jul 17 22:21:14.534135 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:14.534048    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da87ad45-6841-485c-afcc-597845c15442-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5\" (UID: \"da87ad45-6841-485c-afcc-597845c15442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5"
Jul 17 22:21:14.534764 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:14.534744    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da87ad45-6841-485c-afcc-597845c15442-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5\" (UID: \"da87ad45-6841-485c-afcc-597845c15442\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5"
Jul 17 22:21:14.659354 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:14.659262    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5"
Jul 17 22:21:14.798834 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:14.798799    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5"]
Jul 17 22:21:14.802733 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:21:14.802707    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda87ad45_6841_485c_afcc_597845c15442.slice/crio-f99ed963195f0d270bbb0aab3345454c9005963117aa4059f326785f1674a358 WatchSource:0}: Error finding container f99ed963195f0d270bbb0aab3345454c9005963117aa4059f326785f1674a358: Status 404 returned error can't find the container with id f99ed963195f0d270bbb0aab3345454c9005963117aa4059f326785f1674a358
Jul 17 22:21:15.267549 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:15.267512    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5" event={"ID":"da87ad45-6841-485c-afcc-597845c15442","Type":"ContainerStarted","Data":"193a84e3aca7ddfd4e85a0d70a8fb28d324f2942f26a57ea969bc77ae6ec884d"}
Jul 17 22:21:15.267730 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:15.267562    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5" event={"ID":"da87ad45-6841-485c-afcc-597845c15442","Type":"ContainerStarted","Data":"f99ed963195f0d270bbb0aab3345454c9005963117aa4059f326785f1674a358"}
Jul 17 22:21:15.283284 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:15.283227    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5" podStartSLOduration=1.283210362 podStartE2EDuration="1.283210362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:21:15.282207159 +0000 UTC m=+2138.671223820" watchObservedRunningTime="2026-07-17 22:21:15.283210362 +0000 UTC m=+2138.672227321"
Jul 17 22:21:16.272903 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:16.272863    2565 generic.go:356] "Generic (PLEG): container finished" podID="da87ad45-6841-485c-afcc-597845c15442" containerID="193a84e3aca7ddfd4e85a0d70a8fb28d324f2942f26a57ea969bc77ae6ec884d" exitCode=7
Jul 17 22:21:16.273386 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:16.272952    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5" event={"ID":"da87ad45-6841-485c-afcc-597845c15442","Type":"ContainerDied","Data":"193a84e3aca7ddfd4e85a0d70a8fb28d324f2942f26a57ea969bc77ae6ec884d"}
Jul 17 22:21:16.273386 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:16.273168    2565 scope.go:117] "RemoveContainer" containerID="193a84e3aca7ddfd4e85a0d70a8fb28d324f2942f26a57ea969bc77ae6ec884d"
Jul 17 22:21:17.279081 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:17.279021    2565 generic.go:356] "Generic (PLEG): container finished" podID="da87ad45-6841-485c-afcc-597845c15442" containerID="8cf2b5aa3acd5f8af1180f51e20211b194c7298c9299217e610416249a347652" exitCode=7
Jul 17 22:21:17.279607 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:17.279110    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5" event={"ID":"da87ad45-6841-485c-afcc-597845c15442","Type":"ContainerDied","Data":"8cf2b5aa3acd5f8af1180f51e20211b194c7298c9299217e610416249a347652"}
Jul 17 22:21:17.279607 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:17.279186    2565 scope.go:117] "RemoveContainer" containerID="193a84e3aca7ddfd4e85a0d70a8fb28d324f2942f26a57ea969bc77ae6ec884d"
Jul 17 22:21:17.279607 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:17.279542    2565 scope.go:117] "RemoveContainer" containerID="8cf2b5aa3acd5f8af1180f51e20211b194c7298c9299217e610416249a347652"
Jul 17 22:21:17.279836 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:21:17.279797    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5_odh-ai-gateway-infra(da87ad45-6841-485c-afcc-597845c15442)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5" podUID="da87ad45-6841-485c-afcc-597845c15442"
Jul 17 22:21:18.286162 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:18.286134    2565 scope.go:117] "RemoveContainer" containerID="8cf2b5aa3acd5f8af1180f51e20211b194c7298c9299217e610416249a347652"
Jul 17 22:21:18.286534 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:21:18.286335    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5_odh-ai-gateway-infra(da87ad45-6841-485c-afcc-597845c15442)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5" podUID="da87ad45-6841-485c-afcc-597845c15442"
Jul 17 22:21:25.868888 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:25.868846    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb"]
Jul 17 22:21:25.873729 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:25.873709    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb"
Jul 17 22:21:25.879331 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:25.879300    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb"]
Jul 17 22:21:25.934227 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:25.934182    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf9efae-719e-433e-b83c-4f09c88b1bc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb\" (UID: \"fdf9efae-719e-433e-b83c-4f09c88b1bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb"
Jul 17 22:21:26.035709 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:26.035660    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf9efae-719e-433e-b83c-4f09c88b1bc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb\" (UID: \"fdf9efae-719e-433e-b83c-4f09c88b1bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb"
Jul 17 22:21:26.036389 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:26.036360    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf9efae-719e-433e-b83c-4f09c88b1bc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb\" (UID: \"fdf9efae-719e-433e-b83c-4f09c88b1bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb"
Jul 17 22:21:26.186900 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:26.186802    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb"
Jul 17 22:21:26.324159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:26.324118    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb"]
Jul 17 22:21:26.326993 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:21:26.326968    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf9efae_719e_433e_b83c_4f09c88b1bc3.slice/crio-f42a0f7b1cc4b8e063d151e83ffb0632cbc410c694fb463cf2d70bb0c5a966b4 WatchSource:0}: Error finding container f42a0f7b1cc4b8e063d151e83ffb0632cbc410c694fb463cf2d70bb0c5a966b4: Status 404 returned error can't find the container with id f42a0f7b1cc4b8e063d151e83ffb0632cbc410c694fb463cf2d70bb0c5a966b4
Jul 17 22:21:27.178484 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:27.178453    2565 scope.go:117] "RemoveContainer" containerID="c2beaa20eb6a846fa237bf99ac3608098fb8a6dca47e9db8b958c9d57f833fa1"
Jul 17 22:21:27.331808 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:27.331775    2565 generic.go:356] "Generic (PLEG): container finished" podID="fdf9efae-719e-433e-b83c-4f09c88b1bc3" containerID="30946835321fbe8b53457f5a63424332e647b9f6d118f5fb0d5a876a1deec28d" exitCode=6
Jul 17 22:21:27.332006 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:27.331824    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb" event={"ID":"fdf9efae-719e-433e-b83c-4f09c88b1bc3","Type":"ContainerDied","Data":"30946835321fbe8b53457f5a63424332e647b9f6d118f5fb0d5a876a1deec28d"}
Jul 17 22:21:27.332006 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:27.331892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb" event={"ID":"fdf9efae-719e-433e-b83c-4f09c88b1bc3","Type":"ContainerStarted","Data":"f42a0f7b1cc4b8e063d151e83ffb0632cbc410c694fb463cf2d70bb0c5a966b4"}
Jul 17 22:21:27.332126 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:27.332116    2565 scope.go:117] "RemoveContainer" containerID="30946835321fbe8b53457f5a63424332e647b9f6d118f5fb0d5a876a1deec28d"
Jul 17 22:21:28.338229 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:28.338199    2565 generic.go:356] "Generic (PLEG): container finished" podID="a36b1ee2-c4b5-4a8f-8356-7dd5ded76791" containerID="99f27167ae84c7e6cbe329b95704906d21f5fdc55cc877ea6384d1cb73bd95ef" exitCode=6
Jul 17 22:21:28.338681 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:28.338275    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g" event={"ID":"a36b1ee2-c4b5-4a8f-8356-7dd5ded76791","Type":"ContainerDied","Data":"99f27167ae84c7e6cbe329b95704906d21f5fdc55cc877ea6384d1cb73bd95ef"}
Jul 17 22:21:28.338681 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:28.338352    2565 scope.go:117] "RemoveContainer" containerID="c2beaa20eb6a846fa237bf99ac3608098fb8a6dca47e9db8b958c9d57f833fa1"
Jul 17 22:21:28.338681 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:28.338612    2565 scope.go:117] "RemoveContainer" containerID="99f27167ae84c7e6cbe329b95704906d21f5fdc55cc877ea6384d1cb73bd95ef"
Jul 17 22:21:28.338883 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:21:28.338865    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g_odh-ai-gateway-infra(a36b1ee2-c4b5-4a8f-8356-7dd5ded76791)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g" podUID="a36b1ee2-c4b5-4a8f-8356-7dd5ded76791"
Jul 17 22:21:28.340375 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:28.340352    2565 generic.go:356] "Generic (PLEG): container finished" podID="fdf9efae-719e-433e-b83c-4f09c88b1bc3" containerID="0fea03b1a7b0cc7dc193bd3bc347cff1e4a027bde20fa5792bfd596d0d9c5897" exitCode=6
Jul 17 22:21:28.340512 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:28.340429    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb" event={"ID":"fdf9efae-719e-433e-b83c-4f09c88b1bc3","Type":"ContainerDied","Data":"0fea03b1a7b0cc7dc193bd3bc347cff1e4a027bde20fa5792bfd596d0d9c5897"}
Jul 17 22:21:28.340599 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:28.340585    2565 scope.go:117] "RemoveContainer" containerID="0fea03b1a7b0cc7dc193bd3bc347cff1e4a027bde20fa5792bfd596d0d9c5897"
Jul 17 22:21:28.340788 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:21:28.340771    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb_odh-ai-gateway-infra(fdf9efae-719e-433e-b83c-4f09c88b1bc3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb" podUID="fdf9efae-719e-433e-b83c-4f09c88b1bc3"
Jul 17 22:21:28.349897 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:28.349875    2565 scope.go:117] "RemoveContainer" containerID="30946835321fbe8b53457f5a63424332e647b9f6d118f5fb0d5a876a1deec28d"
Jul 17 22:21:29.348529 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:29.348501    2565 scope.go:117] "RemoveContainer" containerID="0fea03b1a7b0cc7dc193bd3bc347cff1e4a027bde20fa5792bfd596d0d9c5897"
Jul 17 22:21:29.348941 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:21:29.348707    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb_odh-ai-gateway-infra(fdf9efae-719e-433e-b83c-4f09c88b1bc3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb" podUID="fdf9efae-719e-433e-b83c-4f09c88b1bc3"
Jul 17 22:21:29.364354 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:29.364321    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g"]
Jul 17 22:21:29.513744 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:29.513717    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g"
Jul 17 22:21:29.568106 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:29.568049    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a36b1ee2-c4b5-4a8f-8356-7dd5ded76791-maas-api-service-ca\") pod \"a36b1ee2-c4b5-4a8f-8356-7dd5ded76791\" (UID: \"a36b1ee2-c4b5-4a8f-8356-7dd5ded76791\") "
Jul 17 22:21:29.568427 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:29.568395    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36b1ee2-c4b5-4a8f-8356-7dd5ded76791-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a36b1ee2-c4b5-4a8f-8356-7dd5ded76791" (UID: "a36b1ee2-c4b5-4a8f-8356-7dd5ded76791"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:21:29.669692 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:29.669598    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a36b1ee2-c4b5-4a8f-8356-7dd5ded76791-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:21:30.354898 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:30.354868    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g"
Jul 17 22:21:30.355395 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:30.354903    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g" event={"ID":"a36b1ee2-c4b5-4a8f-8356-7dd5ded76791","Type":"ContainerDied","Data":"88448d8c5e59624e783591207f49b33b1f8e6193c38930cab0e830eca0687cfe"}
Jul 17 22:21:30.355395 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:30.354963    2565 scope.go:117] "RemoveContainer" containerID="99f27167ae84c7e6cbe329b95704906d21f5fdc55cc877ea6384d1cb73bd95ef"
Jul 17 22:21:30.376946 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:30.376914    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g"]
Jul 17 22:21:30.381415 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:30.381388    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-hg57g"]
Jul 17 22:21:31.176781 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:31.176750    2565 scope.go:117] "RemoveContainer" containerID="8cf2b5aa3acd5f8af1180f51e20211b194c7298c9299217e610416249a347652"
Jul 17 22:21:31.181977 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:31.181950    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a36b1ee2-c4b5-4a8f-8356-7dd5ded76791" path="/var/lib/kubelet/pods/a36b1ee2-c4b5-4a8f-8356-7dd5ded76791/volumes"
Jul 17 22:21:32.367324 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:32.367286    2565 generic.go:356] "Generic (PLEG): container finished" podID="da87ad45-6841-485c-afcc-597845c15442" containerID="4ec49f6be7a84f394698b04e7afb254b0273799c64e03f0d9adf8d9486da8a31" exitCode=7
Jul 17 22:21:32.367754 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:32.367354    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5" event={"ID":"da87ad45-6841-485c-afcc-597845c15442","Type":"ContainerDied","Data":"4ec49f6be7a84f394698b04e7afb254b0273799c64e03f0d9adf8d9486da8a31"}
Jul 17 22:21:32.367754 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:32.367413    2565 scope.go:117] "RemoveContainer" containerID="8cf2b5aa3acd5f8af1180f51e20211b194c7298c9299217e610416249a347652"
Jul 17 22:21:32.367754 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:32.367708    2565 scope.go:117] "RemoveContainer" containerID="4ec49f6be7a84f394698b04e7afb254b0273799c64e03f0d9adf8d9486da8a31"
Jul 17 22:21:32.367972 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:21:32.367950    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5_odh-ai-gateway-infra(da87ad45-6841-485c-afcc-597845c15442)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5" podUID="da87ad45-6841-485c-afcc-597845c15442"
Jul 17 22:21:33.393145 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:33.393113    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5"]
Jul 17 22:21:33.535582 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:33.535558    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5"
Jul 17 22:21:33.603653 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:33.603617    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da87ad45-6841-485c-afcc-597845c15442-maas-api-service-ca\") pod \"da87ad45-6841-485c-afcc-597845c15442\" (UID: \"da87ad45-6841-485c-afcc-597845c15442\") "
Jul 17 22:21:33.604008 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:33.603989    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da87ad45-6841-485c-afcc-597845c15442-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da87ad45-6841-485c-afcc-597845c15442" (UID: "da87ad45-6841-485c-afcc-597845c15442"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:21:33.705201 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:33.705110    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da87ad45-6841-485c-afcc-597845c15442-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:21:34.379934 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:34.379900    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5"
Jul 17 22:21:34.379934 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:34.379919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5" event={"ID":"da87ad45-6841-485c-afcc-597845c15442","Type":"ContainerDied","Data":"f99ed963195f0d270bbb0aab3345454c9005963117aa4059f326785f1674a358"}
Jul 17 22:21:34.380179 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:34.380000    2565 scope.go:117] "RemoveContainer" containerID="4ec49f6be7a84f394698b04e7afb254b0273799c64e03f0d9adf8d9486da8a31"
Jul 17 22:21:34.404144 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:34.404116    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5"]
Jul 17 22:21:34.411840 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:34.411815    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9mxl5"]
Jul 17 22:21:35.181936 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:35.181903    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da87ad45-6841-485c-afcc-597845c15442" path="/var/lib/kubelet/pods/da87ad45-6841-485c-afcc-597845c15442/volumes"
Jul 17 22:21:42.176510 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:42.176479    2565 scope.go:117] "RemoveContainer" containerID="0fea03b1a7b0cc7dc193bd3bc347cff1e4a027bde20fa5792bfd596d0d9c5897"
Jul 17 22:21:43.424990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:43.424958    2565 generic.go:356] "Generic (PLEG): container finished" podID="fdf9efae-719e-433e-b83c-4f09c88b1bc3" containerID="46f7cbed888b65c35df460218efc6e3cc58af95b17534d6a1254db4f76c911b8" exitCode=6
Jul 17 22:21:43.424990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:43.424995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb" event={"ID":"fdf9efae-719e-433e-b83c-4f09c88b1bc3","Type":"ContainerDied","Data":"46f7cbed888b65c35df460218efc6e3cc58af95b17534d6a1254db4f76c911b8"}
Jul 17 22:21:43.425508 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:43.425044    2565 scope.go:117] "RemoveContainer" containerID="0fea03b1a7b0cc7dc193bd3bc347cff1e4a027bde20fa5792bfd596d0d9c5897"
Jul 17 22:21:43.425508 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:43.425315    2565 scope.go:117] "RemoveContainer" containerID="46f7cbed888b65c35df460218efc6e3cc58af95b17534d6a1254db4f76c911b8"
Jul 17 22:21:43.425595 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:21:43.425546    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb_odh-ai-gateway-infra(fdf9efae-719e-433e-b83c-4f09c88b1bc3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb" podUID="fdf9efae-719e-433e-b83c-4f09c88b1bc3"
Jul 17 22:21:44.449525 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:44.449492    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb"]
Jul 17 22:21:44.582995 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:44.582968    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb"
Jul 17 22:21:44.703215 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:44.703121    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf9efae-719e-433e-b83c-4f09c88b1bc3-maas-api-service-ca\") pod \"fdf9efae-719e-433e-b83c-4f09c88b1bc3\" (UID: \"fdf9efae-719e-433e-b83c-4f09c88b1bc3\") "
Jul 17 22:21:44.703489 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:44.703466    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf9efae-719e-433e-b83c-4f09c88b1bc3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fdf9efae-719e-433e-b83c-4f09c88b1bc3" (UID: "fdf9efae-719e-433e-b83c-4f09c88b1bc3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:21:44.803919 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:44.803876    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf9efae-719e-433e-b83c-4f09c88b1bc3-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:21:45.437822 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:45.437794    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb"
Jul 17 22:21:45.438010 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:45.437835    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb" event={"ID":"fdf9efae-719e-433e-b83c-4f09c88b1bc3","Type":"ContainerDied","Data":"f42a0f7b1cc4b8e063d151e83ffb0632cbc410c694fb463cf2d70bb0c5a966b4"}
Jul 17 22:21:45.438010 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:45.437899    2565 scope.go:117] "RemoveContainer" containerID="46f7cbed888b65c35df460218efc6e3cc58af95b17534d6a1254db4f76c911b8"
Jul 17 22:21:45.454635 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:45.454606    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb"]
Jul 17 22:21:45.458241 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:45.458215    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-fsxxb"]
Jul 17 22:21:47.183723 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:21:47.183689    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdf9efae-719e-433e-b83c-4f09c88b1bc3" path="/var/lib/kubelet/pods/fdf9efae-719e-433e-b83c-4f09c88b1bc3/volumes"
Jul 17 22:22:07.712631 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.712596    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk"]
Jul 17 22:22:07.713096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.712948    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf9efae-719e-433e-b83c-4f09c88b1bc3" containerName="revoke-keys"
Jul 17 22:22:07.713096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.712960    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf9efae-719e-433e-b83c-4f09c88b1bc3" containerName="revoke-keys"
Jul 17 22:22:07.713096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.712969    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a36b1ee2-c4b5-4a8f-8356-7dd5ded76791" containerName="revoke-keys"
Jul 17 22:22:07.713096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.712974    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a36b1ee2-c4b5-4a8f-8356-7dd5ded76791" containerName="revoke-keys"
Jul 17 22:22:07.713096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.712988    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf9efae-719e-433e-b83c-4f09c88b1bc3" containerName="revoke-keys"
Jul 17 22:22:07.713096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.712994    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf9efae-719e-433e-b83c-4f09c88b1bc3" containerName="revoke-keys"
Jul 17 22:22:07.713096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.712999    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a36b1ee2-c4b5-4a8f-8356-7dd5ded76791" containerName="revoke-keys"
Jul 17 22:22:07.713096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.713005    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a36b1ee2-c4b5-4a8f-8356-7dd5ded76791" containerName="revoke-keys"
Jul 17 22:22:07.713096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.713011    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da87ad45-6841-485c-afcc-597845c15442" containerName="revoke-keys"
Jul 17 22:22:07.713096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.713016    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da87ad45-6841-485c-afcc-597845c15442" containerName="revoke-keys"
Jul 17 22:22:07.713096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.713023    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da87ad45-6841-485c-afcc-597845c15442" containerName="revoke-keys"
Jul 17 22:22:07.713096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.713028    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da87ad45-6841-485c-afcc-597845c15442" containerName="revoke-keys"
Jul 17 22:22:07.713096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.713038    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf9efae-719e-433e-b83c-4f09c88b1bc3" containerName="revoke-keys"
Jul 17 22:22:07.713096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.713043    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf9efae-719e-433e-b83c-4f09c88b1bc3" containerName="revoke-keys"
Jul 17 22:22:07.713578 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.713132    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf9efae-719e-433e-b83c-4f09c88b1bc3" containerName="revoke-keys"
Jul 17 22:22:07.713578 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.713140    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a36b1ee2-c4b5-4a8f-8356-7dd5ded76791" containerName="revoke-keys"
Jul 17 22:22:07.713578 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.713146    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a36b1ee2-c4b5-4a8f-8356-7dd5ded76791" containerName="revoke-keys"
Jul 17 22:22:07.713578 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.713154    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da87ad45-6841-485c-afcc-597845c15442" containerName="revoke-keys"
Jul 17 22:22:07.713578 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.713160    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da87ad45-6841-485c-afcc-597845c15442" containerName="revoke-keys"
Jul 17 22:22:07.713578 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.713166    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf9efae-719e-433e-b83c-4f09c88b1bc3" containerName="revoke-keys"
Jul 17 22:22:07.713578 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.713172    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da87ad45-6841-485c-afcc-597845c15442" containerName="revoke-keys"
Jul 17 22:22:07.717559 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.717539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk"
Jul 17 22:22:07.720930 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.720903    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:22:07.721109 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.720961    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:22:07.722800 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.722768    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk"]
Jul 17 22:22:07.812736 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.812696    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdedc97a-2a46-422c-bc72-1031ebee2871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk\" (UID: \"fdedc97a-2a46-422c-bc72-1031ebee2871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk"
Jul 17 22:22:07.913273 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.913236    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdedc97a-2a46-422c-bc72-1031ebee2871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk\" (UID: \"fdedc97a-2a46-422c-bc72-1031ebee2871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk"
Jul 17 22:22:07.913957 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:07.913932    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdedc97a-2a46-422c-bc72-1031ebee2871-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk\" (UID: \"fdedc97a-2a46-422c-bc72-1031ebee2871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk"
Jul 17 22:22:08.030339 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:08.030305    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk"
Jul 17 22:22:08.170526 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:08.170498    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk"]
Jul 17 22:22:08.173350 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:22:08.173323    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdedc97a_2a46_422c_bc72_1031ebee2871.slice/crio-9c0a3cd3e0d1ab689960d39d2ca4284fb343ca542b285cae2cb22dd4d3ea5182 WatchSource:0}: Error finding container 9c0a3cd3e0d1ab689960d39d2ca4284fb343ca542b285cae2cb22dd4d3ea5182: Status 404 returned error can't find the container with id 9c0a3cd3e0d1ab689960d39d2ca4284fb343ca542b285cae2cb22dd4d3ea5182
Jul 17 22:22:08.548360 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:08.548329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk" event={"ID":"fdedc97a-2a46-422c-bc72-1031ebee2871","Type":"ContainerStarted","Data":"0ec3c685247cfddc104ce144ad7f3a5721e64a94ce74ac483c8fc756d2e64f91"}
Jul 17 22:22:08.548541 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:08.548380    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk" event={"ID":"fdedc97a-2a46-422c-bc72-1031ebee2871","Type":"ContainerStarted","Data":"9c0a3cd3e0d1ab689960d39d2ca4284fb343ca542b285cae2cb22dd4d3ea5182"}
Jul 17 22:22:08.567797 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:08.567743    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk" podStartSLOduration=1.567727776 podStartE2EDuration="1.567727776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:22:08.565255014 +0000 UTC m=+2191.954271678" watchObservedRunningTime="2026-07-17 22:22:08.567727776 +0000 UTC m=+2191.956744427"
Jul 17 22:22:09.554866 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:09.554828    2565 generic.go:356] "Generic (PLEG): container finished" podID="fdedc97a-2a46-422c-bc72-1031ebee2871" containerID="0ec3c685247cfddc104ce144ad7f3a5721e64a94ce74ac483c8fc756d2e64f91" exitCode=6
Jul 17 22:22:09.555352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:09.554918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk" event={"ID":"fdedc97a-2a46-422c-bc72-1031ebee2871","Type":"ContainerDied","Data":"0ec3c685247cfddc104ce144ad7f3a5721e64a94ce74ac483c8fc756d2e64f91"}
Jul 17 22:22:09.555352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:09.555099    2565 scope.go:117] "RemoveContainer" containerID="0ec3c685247cfddc104ce144ad7f3a5721e64a94ce74ac483c8fc756d2e64f91"
Jul 17 22:22:10.561344 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:10.561303    2565 generic.go:356] "Generic (PLEG): container finished" podID="fdedc97a-2a46-422c-bc72-1031ebee2871" containerID="6e9162a46b4170b34efa40ab6b2f5cfb851f864db5b8bdbcdf0bc4a6ada552da" exitCode=6
Jul 17 22:22:10.561731 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:10.561360    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk" event={"ID":"fdedc97a-2a46-422c-bc72-1031ebee2871","Type":"ContainerDied","Data":"6e9162a46b4170b34efa40ab6b2f5cfb851f864db5b8bdbcdf0bc4a6ada552da"}
Jul 17 22:22:10.561731 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:10.561419    2565 scope.go:117] "RemoveContainer" containerID="0ec3c685247cfddc104ce144ad7f3a5721e64a94ce74ac483c8fc756d2e64f91"
Jul 17 22:22:10.561731 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:10.561576    2565 scope.go:117] "RemoveContainer" containerID="6e9162a46b4170b34efa40ab6b2f5cfb851f864db5b8bdbcdf0bc4a6ada552da"
Jul 17 22:22:10.561883 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:22:10.561818    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk_odh-ai-gateway-infra(fdedc97a-2a46-422c-bc72-1031ebee2871)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk" podUID="fdedc97a-2a46-422c-bc72-1031ebee2871"
Jul 17 22:22:11.568181 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:11.568148    2565 scope.go:117] "RemoveContainer" containerID="6e9162a46b4170b34efa40ab6b2f5cfb851f864db5b8bdbcdf0bc4a6ada552da"
Jul 17 22:22:11.568608 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:22:11.568330    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk_odh-ai-gateway-infra(fdedc97a-2a46-422c-bc72-1031ebee2871)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk" podUID="fdedc97a-2a46-422c-bc72-1031ebee2871"
Jul 17 22:22:14.391452 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:14.391405    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh"]
Jul 17 22:22:14.392103 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:14.391939    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a36b1ee2-c4b5-4a8f-8356-7dd5ded76791" containerName="revoke-keys"
Jul 17 22:22:14.392103 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:14.391961    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a36b1ee2-c4b5-4a8f-8356-7dd5ded76791" containerName="revoke-keys"
Jul 17 22:22:14.392103 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:14.391977    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da87ad45-6841-485c-afcc-597845c15442" containerName="revoke-keys"
Jul 17 22:22:14.392103 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:14.391986    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da87ad45-6841-485c-afcc-597845c15442" containerName="revoke-keys"
Jul 17 22:22:14.392103 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:14.392091    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf9efae-719e-433e-b83c-4f09c88b1bc3" containerName="revoke-keys"
Jul 17 22:22:14.392103 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:14.392107    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a36b1ee2-c4b5-4a8f-8356-7dd5ded76791" containerName="revoke-keys"
Jul 17 22:22:14.396297 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:14.396273    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh"
Jul 17 22:22:14.400864 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:14.400834    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh"]
Jul 17 22:22:14.470887 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:14.470843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f0adff-9cb7-4a54-8d3f-d523555a5eca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh\" (UID: \"d5f0adff-9cb7-4a54-8d3f-d523555a5eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh"
Jul 17 22:22:14.571507 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:14.571466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f0adff-9cb7-4a54-8d3f-d523555a5eca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh\" (UID: \"d5f0adff-9cb7-4a54-8d3f-d523555a5eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh"
Jul 17 22:22:14.572106 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:14.572056    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f0adff-9cb7-4a54-8d3f-d523555a5eca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh\" (UID: \"d5f0adff-9cb7-4a54-8d3f-d523555a5eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh"
Jul 17 22:22:14.709957 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:14.709868    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh"
Jul 17 22:22:14.848930 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:14.848903    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh"]
Jul 17 22:22:14.850740 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:22:14.850708    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f0adff_9cb7_4a54_8d3f_d523555a5eca.slice/crio-69ee9d78c5dc2f8f9870f11a84a7bc0659a9bab2df2a305dd98d82b090ce7194 WatchSource:0}: Error finding container 69ee9d78c5dc2f8f9870f11a84a7bc0659a9bab2df2a305dd98d82b090ce7194: Status 404 returned error can't find the container with id 69ee9d78c5dc2f8f9870f11a84a7bc0659a9bab2df2a305dd98d82b090ce7194
Jul 17 22:22:15.587990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:15.587950    2565 generic.go:356] "Generic (PLEG): container finished" podID="d5f0adff-9cb7-4a54-8d3f-d523555a5eca" containerID="41e29f20f9d659054e240126295f22761007f8cb866ce8d8631a15b0096870e0" exitCode=7
Jul 17 22:22:15.588497 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:15.588037    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh" event={"ID":"d5f0adff-9cb7-4a54-8d3f-d523555a5eca","Type":"ContainerDied","Data":"41e29f20f9d659054e240126295f22761007f8cb866ce8d8631a15b0096870e0"}
Jul 17 22:22:15.588497 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:15.588118    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh" event={"ID":"d5f0adff-9cb7-4a54-8d3f-d523555a5eca","Type":"ContainerStarted","Data":"69ee9d78c5dc2f8f9870f11a84a7bc0659a9bab2df2a305dd98d82b090ce7194"}
Jul 17 22:22:15.588497 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:15.588247    2565 scope.go:117] "RemoveContainer" containerID="41e29f20f9d659054e240126295f22761007f8cb866ce8d8631a15b0096870e0"
Jul 17 22:22:15.907636 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:15.907600    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl"]
Jul 17 22:22:15.911122 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:15.911101    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl"
Jul 17 22:22:15.918374 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:15.918340    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl"]
Jul 17 22:22:15.985214 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:15.985172    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9401138f-6f70-4608-8873-84ab87ca77eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl\" (UID: \"9401138f-6f70-4608-8873-84ab87ca77eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl"
Jul 17 22:22:16.085977 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:16.085946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9401138f-6f70-4608-8873-84ab87ca77eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl\" (UID: \"9401138f-6f70-4608-8873-84ab87ca77eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl"
Jul 17 22:22:16.086629 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:16.086606    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9401138f-6f70-4608-8873-84ab87ca77eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl\" (UID: \"9401138f-6f70-4608-8873-84ab87ca77eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl"
Jul 17 22:22:16.223620 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:16.223526    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl"
Jul 17 22:22:16.364546 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:16.364518    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl"]
Jul 17 22:22:16.366781 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:22:16.366748    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9401138f_6f70_4608_8873_84ab87ca77eb.slice/crio-fc3dff57d296357608718945e33b11401d713e37b33e7f2b382025eb6a96a3a3 WatchSource:0}: Error finding container fc3dff57d296357608718945e33b11401d713e37b33e7f2b382025eb6a96a3a3: Status 404 returned error can't find the container with id fc3dff57d296357608718945e33b11401d713e37b33e7f2b382025eb6a96a3a3
Jul 17 22:22:16.594904 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:16.594864    2565 generic.go:356] "Generic (PLEG): container finished" podID="d5f0adff-9cb7-4a54-8d3f-d523555a5eca" containerID="851b33361ab12d5af2f75a193ecd797ff5140426153ac0434a8c6fc66a92c5bd" exitCode=7
Jul 17 22:22:16.595357 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:16.594921    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh" event={"ID":"d5f0adff-9cb7-4a54-8d3f-d523555a5eca","Type":"ContainerDied","Data":"851b33361ab12d5af2f75a193ecd797ff5140426153ac0434a8c6fc66a92c5bd"}
Jul 17 22:22:16.595357 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:16.594983    2565 scope.go:117] "RemoveContainer" containerID="41e29f20f9d659054e240126295f22761007f8cb866ce8d8631a15b0096870e0"
Jul 17 22:22:16.595357 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:16.595205    2565 scope.go:117] "RemoveContainer" containerID="851b33361ab12d5af2f75a193ecd797ff5140426153ac0434a8c6fc66a92c5bd"
Jul 17 22:22:16.595499 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:22:16.595464    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh_odh-ai-gateway-infra(d5f0adff-9cb7-4a54-8d3f-d523555a5eca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh" podUID="d5f0adff-9cb7-4a54-8d3f-d523555a5eca"
Jul 17 22:22:16.596261 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:16.596237    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl" event={"ID":"9401138f-6f70-4608-8873-84ab87ca77eb","Type":"ContainerStarted","Data":"fc3dff57d296357608718945e33b11401d713e37b33e7f2b382025eb6a96a3a3"}
Jul 17 22:22:17.602810 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:17.602781    2565 scope.go:117] "RemoveContainer" containerID="851b33361ab12d5af2f75a193ecd797ff5140426153ac0434a8c6fc66a92c5bd"
Jul 17 22:22:17.603286 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:22:17.603016    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh_odh-ai-gateway-infra(d5f0adff-9cb7-4a54-8d3f-d523555a5eca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh" podUID="d5f0adff-9cb7-4a54-8d3f-d523555a5eca"
Jul 17 22:22:17.604131 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:17.604106    2565 generic.go:356] "Generic (PLEG): container finished" podID="9401138f-6f70-4608-8873-84ab87ca77eb" containerID="fe19ec34f7201b9d0030c95bb3cf72527555e9feb847d766eca044e6006bb8d7" exitCode=6
Jul 17 22:22:17.604252 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:17.604197    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl" event={"ID":"9401138f-6f70-4608-8873-84ab87ca77eb","Type":"ContainerDied","Data":"fe19ec34f7201b9d0030c95bb3cf72527555e9feb847d766eca044e6006bb8d7"}
Jul 17 22:22:17.604341 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:17.604328    2565 scope.go:117] "RemoveContainer" containerID="fe19ec34f7201b9d0030c95bb3cf72527555e9feb847d766eca044e6006bb8d7"
Jul 17 22:22:18.610677 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:18.610568    2565 generic.go:356] "Generic (PLEG): container finished" podID="9401138f-6f70-4608-8873-84ab87ca77eb" containerID="14fb63ad96ac2bfdb29643aa0404edbfce8791e90f71e0dd9f2f206e6e68e9dd" exitCode=6
Jul 17 22:22:18.610677 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:18.610652    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl" event={"ID":"9401138f-6f70-4608-8873-84ab87ca77eb","Type":"ContainerDied","Data":"14fb63ad96ac2bfdb29643aa0404edbfce8791e90f71e0dd9f2f206e6e68e9dd"}
Jul 17 22:22:18.611131 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:18.610722    2565 scope.go:117] "RemoveContainer" containerID="fe19ec34f7201b9d0030c95bb3cf72527555e9feb847d766eca044e6006bb8d7"
Jul 17 22:22:18.611131 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:18.610862    2565 scope.go:117] "RemoveContainer" containerID="14fb63ad96ac2bfdb29643aa0404edbfce8791e90f71e0dd9f2f206e6e68e9dd"
Jul 17 22:22:18.611131 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:22:18.611089    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl_odh-ai-gateway-infra(9401138f-6f70-4608-8873-84ab87ca77eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl" podUID="9401138f-6f70-4608-8873-84ab87ca77eb"
Jul 17 22:22:19.617795 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:19.617766    2565 scope.go:117] "RemoveContainer" containerID="14fb63ad96ac2bfdb29643aa0404edbfce8791e90f71e0dd9f2f206e6e68e9dd"
Jul 17 22:22:19.618253 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:22:19.617988    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl_odh-ai-gateway-infra(9401138f-6f70-4608-8873-84ab87ca77eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl" podUID="9401138f-6f70-4608-8873-84ab87ca77eb"
Jul 17 22:22:19.819192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:19.819152    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:22:22.177152 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:22.177120    2565 scope.go:117] "RemoveContainer" containerID="6e9162a46b4170b34efa40ab6b2f5cfb851f864db5b8bdbcdf0bc4a6ada552da"
Jul 17 22:22:22.634721 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:22.634685    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk" event={"ID":"fdedc97a-2a46-422c-bc72-1031ebee2871","Type":"ContainerStarted","Data":"2c635b2fd886a69a9b34d46c70cb352e6979e6b3478c4d5b5cbe668045e9cccd"}
Jul 17 22:22:23.641394 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:23.641358    2565 generic.go:356] "Generic (PLEG): container finished" podID="fdedc97a-2a46-422c-bc72-1031ebee2871" containerID="2c635b2fd886a69a9b34d46c70cb352e6979e6b3478c4d5b5cbe668045e9cccd" exitCode=6
Jul 17 22:22:23.641848 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:23.641406    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk" event={"ID":"fdedc97a-2a46-422c-bc72-1031ebee2871","Type":"ContainerDied","Data":"2c635b2fd886a69a9b34d46c70cb352e6979e6b3478c4d5b5cbe668045e9cccd"}
Jul 17 22:22:23.641848 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:23.641461    2565 scope.go:117] "RemoveContainer" containerID="6e9162a46b4170b34efa40ab6b2f5cfb851f864db5b8bdbcdf0bc4a6ada552da"
Jul 17 22:22:23.641848 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:23.641784    2565 scope.go:117] "RemoveContainer" containerID="2c635b2fd886a69a9b34d46c70cb352e6979e6b3478c4d5b5cbe668045e9cccd"
Jul 17 22:22:23.642038 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:22:23.642017    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk_odh-ai-gateway-infra(fdedc97a-2a46-422c-bc72-1031ebee2871)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk" podUID="fdedc97a-2a46-422c-bc72-1031ebee2871"
Jul 17 22:22:23.662203 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:23.662174    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk"]
Jul 17 22:22:24.792439 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:24.792411    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk"
Jul 17 22:22:24.864581 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:24.864539    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdedc97a-2a46-422c-bc72-1031ebee2871-maas-api-service-ca\") pod \"fdedc97a-2a46-422c-bc72-1031ebee2871\" (UID: \"fdedc97a-2a46-422c-bc72-1031ebee2871\") "
Jul 17 22:22:24.864946 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:24.864920    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdedc97a-2a46-422c-bc72-1031ebee2871-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fdedc97a-2a46-422c-bc72-1031ebee2871" (UID: "fdedc97a-2a46-422c-bc72-1031ebee2871"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:22:24.965584 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:24.965492    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdedc97a-2a46-422c-bc72-1031ebee2871-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:22:25.653212 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:25.653179    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk"
Jul 17 22:22:25.653212 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:25.653204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk" event={"ID":"fdedc97a-2a46-422c-bc72-1031ebee2871","Type":"ContainerDied","Data":"9c0a3cd3e0d1ab689960d39d2ca4284fb343ca542b285cae2cb22dd4d3ea5182"}
Jul 17 22:22:25.653478 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:25.653271    2565 scope.go:117] "RemoveContainer" containerID="2c635b2fd886a69a9b34d46c70cb352e6979e6b3478c4d5b5cbe668045e9cccd"
Jul 17 22:22:25.671991 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:25.671951    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk"]
Jul 17 22:22:25.682260 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:25.682229    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-w9cvk"]
Jul 17 22:22:27.182384 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:27.182341    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdedc97a-2a46-422c-bc72-1031ebee2871" path="/var/lib/kubelet/pods/fdedc97a-2a46-422c-bc72-1031ebee2871/volumes"
Jul 17 22:22:30.177353 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:30.177317    2565 scope.go:117] "RemoveContainer" containerID="851b33361ab12d5af2f75a193ecd797ff5140426153ac0434a8c6fc66a92c5bd"
Jul 17 22:22:30.679529 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:30.679494    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh" event={"ID":"d5f0adff-9cb7-4a54-8d3f-d523555a5eca","Type":"ContainerStarted","Data":"6a70952841f3264c91ac4cb9b1cb542b4965f0e4c51ec5ca55c099d1db35b81f"}
Jul 17 22:22:30.702455 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:30.702391    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh" podStartSLOduration=16.702373401 podStartE2EDuration="16.702373401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:22:30.700937944 +0000 UTC m=+2214.089954593" watchObservedRunningTime="2026-07-17 22:22:30.702373401 +0000 UTC m=+2214.091390030"
Jul 17 22:22:31.685947 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:31.685913    2565 generic.go:356] "Generic (PLEG): container finished" podID="d5f0adff-9cb7-4a54-8d3f-d523555a5eca" containerID="6a70952841f3264c91ac4cb9b1cb542b4965f0e4c51ec5ca55c099d1db35b81f" exitCode=7
Jul 17 22:22:31.685947 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:31.685956    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh" event={"ID":"d5f0adff-9cb7-4a54-8d3f-d523555a5eca","Type":"ContainerDied","Data":"6a70952841f3264c91ac4cb9b1cb542b4965f0e4c51ec5ca55c099d1db35b81f"}
Jul 17 22:22:31.686450 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:31.686010    2565 scope.go:117] "RemoveContainer" containerID="851b33361ab12d5af2f75a193ecd797ff5140426153ac0434a8c6fc66a92c5bd"
Jul 17 22:22:31.686450 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:31.686353    2565 scope.go:117] "RemoveContainer" containerID="6a70952841f3264c91ac4cb9b1cb542b4965f0e4c51ec5ca55c099d1db35b81f"
Jul 17 22:22:31.686620 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:22:31.686600    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh_odh-ai-gateway-infra(d5f0adff-9cb7-4a54-8d3f-d523555a5eca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh" podUID="d5f0adff-9cb7-4a54-8d3f-d523555a5eca"
Jul 17 22:22:31.728794 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:31.728755    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh"]
Jul 17 22:22:32.828201 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:32.828174    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh"
Jul 17 22:22:32.940793 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:32.940751    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f0adff-9cb7-4a54-8d3f-d523555a5eca-maas-api-service-ca\") pod \"d5f0adff-9cb7-4a54-8d3f-d523555a5eca\" (UID: \"d5f0adff-9cb7-4a54-8d3f-d523555a5eca\") "
Jul 17 22:22:32.941176 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:32.941154    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f0adff-9cb7-4a54-8d3f-d523555a5eca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d5f0adff-9cb7-4a54-8d3f-d523555a5eca" (UID: "d5f0adff-9cb7-4a54-8d3f-d523555a5eca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:22:33.041369 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:33.041338    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f0adff-9cb7-4a54-8d3f-d523555a5eca-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:22:33.176673 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:33.176638    2565 scope.go:117] "RemoveContainer" containerID="14fb63ad96ac2bfdb29643aa0404edbfce8791e90f71e0dd9f2f206e6e68e9dd"
Jul 17 22:22:33.698520 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:33.698407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl" event={"ID":"9401138f-6f70-4608-8873-84ab87ca77eb","Type":"ContainerStarted","Data":"65e9206aa9a26a86137a52a6a857c279c3e867e5d49f35c5d5607ebeb6dec801"}
Jul 17 22:22:33.699692 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:33.699666    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh" event={"ID":"d5f0adff-9cb7-4a54-8d3f-d523555a5eca","Type":"ContainerDied","Data":"69ee9d78c5dc2f8f9870f11a84a7bc0659a9bab2df2a305dd98d82b090ce7194"}
Jul 17 22:22:33.699692 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:33.699693    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh"
Jul 17 22:22:33.699890 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:33.699723    2565 scope.go:117] "RemoveContainer" containerID="6a70952841f3264c91ac4cb9b1cb542b4965f0e4c51ec5ca55c099d1db35b81f"
Jul 17 22:22:33.715951 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:33.715901    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl" podStartSLOduration=18.715881672 podStartE2EDuration="18.715881672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:22:33.713524392 +0000 UTC m=+2217.102541040" watchObservedRunningTime="2026-07-17 22:22:33.715881672 +0000 UTC m=+2217.104898323"
Jul 17 22:22:33.726542 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:33.726511    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh"]
Jul 17 22:22:33.730157 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:33.730118    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-m8xlh"]
Jul 17 22:22:34.707018 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:34.706976    2565 generic.go:356] "Generic (PLEG): container finished" podID="9401138f-6f70-4608-8873-84ab87ca77eb" containerID="65e9206aa9a26a86137a52a6a857c279c3e867e5d49f35c5d5607ebeb6dec801" exitCode=6
Jul 17 22:22:34.707480 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:34.707051    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl" event={"ID":"9401138f-6f70-4608-8873-84ab87ca77eb","Type":"ContainerDied","Data":"65e9206aa9a26a86137a52a6a857c279c3e867e5d49f35c5d5607ebeb6dec801"}
Jul 17 22:22:34.707480 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:34.707136    2565 scope.go:117] "RemoveContainer" containerID="14fb63ad96ac2bfdb29643aa0404edbfce8791e90f71e0dd9f2f206e6e68e9dd"
Jul 17 22:22:34.707480 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:34.707447    2565 scope.go:117] "RemoveContainer" containerID="65e9206aa9a26a86137a52a6a857c279c3e867e5d49f35c5d5607ebeb6dec801"
Jul 17 22:22:34.707698 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:22:34.707674    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl_odh-ai-gateway-infra(9401138f-6f70-4608-8873-84ab87ca77eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl" podUID="9401138f-6f70-4608-8873-84ab87ca77eb"
Jul 17 22:22:34.723368 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:34.723340    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl"]
Jul 17 22:22:35.182483 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:35.182444    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5f0adff-9cb7-4a54-8d3f-d523555a5eca" path="/var/lib/kubelet/pods/d5f0adff-9cb7-4a54-8d3f-d523555a5eca/volumes"
Jul 17 22:22:35.853757 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:35.853733    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl"
Jul 17 22:22:35.967116 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:35.966985    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9401138f-6f70-4608-8873-84ab87ca77eb-maas-api-service-ca\") pod \"9401138f-6f70-4608-8873-84ab87ca77eb\" (UID: \"9401138f-6f70-4608-8873-84ab87ca77eb\") "
Jul 17 22:22:35.967374 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:35.967352    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9401138f-6f70-4608-8873-84ab87ca77eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9401138f-6f70-4608-8873-84ab87ca77eb" (UID: "9401138f-6f70-4608-8873-84ab87ca77eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:22:36.068281 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:36.068244    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9401138f-6f70-4608-8873-84ab87ca77eb-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:22:36.725757 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:36.725725    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl"
Jul 17 22:22:36.725757 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:36.725736    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl" event={"ID":"9401138f-6f70-4608-8873-84ab87ca77eb","Type":"ContainerDied","Data":"fc3dff57d296357608718945e33b11401d713e37b33e7f2b382025eb6a96a3a3"}
Jul 17 22:22:36.726029 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:36.725829    2565 scope.go:117] "RemoveContainer" containerID="65e9206aa9a26a86137a52a6a857c279c3e867e5d49f35c5d5607ebeb6dec801"
Jul 17 22:22:36.748342 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:36.748313    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl"]
Jul 17 22:22:36.752049 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:36.752023    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-8g7vl"]
Jul 17 22:22:37.182462 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:37.182423    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9401138f-6f70-4608-8873-84ab87ca77eb" path="/var/lib/kubelet/pods/9401138f-6f70-4608-8873-84ab87ca77eb/volumes"
Jul 17 22:22:57.760453 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.759879    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq"]
Jul 17 22:22:57.761777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.761328    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdedc97a-2a46-422c-bc72-1031ebee2871" containerName="revoke-keys"
Jul 17 22:22:57.761777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.761354    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdedc97a-2a46-422c-bc72-1031ebee2871" containerName="revoke-keys"
Jul 17 22:22:57.761777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.761372    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f0adff-9cb7-4a54-8d3f-d523555a5eca" containerName="revoke-keys"
Jul 17 22:22:57.761777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.761382    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f0adff-9cb7-4a54-8d3f-d523555a5eca" containerName="revoke-keys"
Jul 17 22:22:57.761777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.761391    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9401138f-6f70-4608-8873-84ab87ca77eb" containerName="revoke-keys"
Jul 17 22:22:57.761777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.761399    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9401138f-6f70-4608-8873-84ab87ca77eb" containerName="revoke-keys"
Jul 17 22:22:57.761777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.761425    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f0adff-9cb7-4a54-8d3f-d523555a5eca" containerName="revoke-keys"
Jul 17 22:22:57.761777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.761433    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f0adff-9cb7-4a54-8d3f-d523555a5eca" containerName="revoke-keys"
Jul 17 22:22:57.761777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.761449    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdedc97a-2a46-422c-bc72-1031ebee2871" containerName="revoke-keys"
Jul 17 22:22:57.761777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.761457    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdedc97a-2a46-422c-bc72-1031ebee2871" containerName="revoke-keys"
Jul 17 22:22:57.761777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.761481    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f0adff-9cb7-4a54-8d3f-d523555a5eca" containerName="revoke-keys"
Jul 17 22:22:57.764351 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.762390    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f0adff-9cb7-4a54-8d3f-d523555a5eca" containerName="revoke-keys"
Jul 17 22:22:57.764351 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.762435    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9401138f-6f70-4608-8873-84ab87ca77eb" containerName="revoke-keys"
Jul 17 22:22:57.764351 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.762444    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9401138f-6f70-4608-8873-84ab87ca77eb" containerName="revoke-keys"
Jul 17 22:22:57.764351 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.762634    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f0adff-9cb7-4a54-8d3f-d523555a5eca" containerName="revoke-keys"
Jul 17 22:22:57.764351 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.762652    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f0adff-9cb7-4a54-8d3f-d523555a5eca" containerName="revoke-keys"
Jul 17 22:22:57.764351 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.762664    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdedc97a-2a46-422c-bc72-1031ebee2871" containerName="revoke-keys"
Jul 17 22:22:57.764351 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.762680    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdedc97a-2a46-422c-bc72-1031ebee2871" containerName="revoke-keys"
Jul 17 22:22:57.764351 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.762689    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f0adff-9cb7-4a54-8d3f-d523555a5eca" containerName="revoke-keys"
Jul 17 22:22:57.764351 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.762698    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9401138f-6f70-4608-8873-84ab87ca77eb" containerName="revoke-keys"
Jul 17 22:22:57.764351 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.762710    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9401138f-6f70-4608-8873-84ab87ca77eb" containerName="revoke-keys"
Jul 17 22:22:57.764351 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.762724    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9401138f-6f70-4608-8873-84ab87ca77eb" containerName="revoke-keys"
Jul 17 22:22:57.766667 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.766645    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq"
Jul 17 22:22:57.769739 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.769701    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:22:57.769969 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.769728    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:22:57.771789 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.771763    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq"]
Jul 17 22:22:57.875470 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.875432    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b42415c-9fc4-44e2-868f-003043394609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq\" (UID: \"7b42415c-9fc4-44e2-868f-003043394609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq"
Jul 17 22:22:57.976028 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.975980    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b42415c-9fc4-44e2-868f-003043394609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq\" (UID: \"7b42415c-9fc4-44e2-868f-003043394609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq"
Jul 17 22:22:57.976627 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:57.976606    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b42415c-9fc4-44e2-868f-003043394609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq\" (UID: \"7b42415c-9fc4-44e2-868f-003043394609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq"
Jul 17 22:22:58.081712 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:58.081622    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq"
Jul 17 22:22:58.223522 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:58.223416    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq"]
Jul 17 22:22:58.226375 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:22:58.226330    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b42415c_9fc4_44e2_868f_003043394609.slice/crio-1fd9797e51f1d4337c0414831fa6974553abacb0d872861595f81605a122f096 WatchSource:0}: Error finding container 1fd9797e51f1d4337c0414831fa6974553abacb0d872861595f81605a122f096: Status 404 returned error can't find the container with id 1fd9797e51f1d4337c0414831fa6974553abacb0d872861595f81605a122f096
Jul 17 22:22:58.827878 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:58.827842    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq" event={"ID":"7b42415c-9fc4-44e2-868f-003043394609","Type":"ContainerStarted","Data":"5e17d52c9b6f5ba4f940ed634ddbdd1189a3cc11f29a1a973ec53c31ff600910"}
Jul 17 22:22:58.828340 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:58.827896    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq" event={"ID":"7b42415c-9fc4-44e2-868f-003043394609","Type":"ContainerStarted","Data":"1fd9797e51f1d4337c0414831fa6974553abacb0d872861595f81605a122f096"}
Jul 17 22:22:58.844878 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:58.844816    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq" podStartSLOduration=1.844798205 podStartE2EDuration="1.844798205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:22:58.843034314 +0000 UTC m=+2242.232050966" watchObservedRunningTime="2026-07-17 22:22:58.844798205 +0000 UTC m=+2242.233814856"
Jul 17 22:22:59.834505 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:59.834470    2565 generic.go:356] "Generic (PLEG): container finished" podID="7b42415c-9fc4-44e2-868f-003043394609" containerID="5e17d52c9b6f5ba4f940ed634ddbdd1189a3cc11f29a1a973ec53c31ff600910" exitCode=6
Jul 17 22:22:59.834906 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:59.834552    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq" event={"ID":"7b42415c-9fc4-44e2-868f-003043394609","Type":"ContainerDied","Data":"5e17d52c9b6f5ba4f940ed634ddbdd1189a3cc11f29a1a973ec53c31ff600910"}
Jul 17 22:22:59.834906 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:22:59.834731    2565 scope.go:117] "RemoveContainer" containerID="5e17d52c9b6f5ba4f940ed634ddbdd1189a3cc11f29a1a973ec53c31ff600910"
Jul 17 22:23:00.841257 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:00.841169    2565 generic.go:356] "Generic (PLEG): container finished" podID="7b42415c-9fc4-44e2-868f-003043394609" containerID="15c0b79c530f246d3effb7e3f0c4833633db17a348ea45ef0c995749ff56a93f" exitCode=6
Jul 17 22:23:00.841257 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:00.841218    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq" event={"ID":"7b42415c-9fc4-44e2-868f-003043394609","Type":"ContainerDied","Data":"15c0b79c530f246d3effb7e3f0c4833633db17a348ea45ef0c995749ff56a93f"}
Jul 17 22:23:00.841726 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:00.841267    2565 scope.go:117] "RemoveContainer" containerID="5e17d52c9b6f5ba4f940ed634ddbdd1189a3cc11f29a1a973ec53c31ff600910"
Jul 17 22:23:00.841726 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:00.841463    2565 scope.go:117] "RemoveContainer" containerID="15c0b79c530f246d3effb7e3f0c4833633db17a348ea45ef0c995749ff56a93f"
Jul 17 22:23:00.841726 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:23:00.841707    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq_odh-ai-gateway-infra(7b42415c-9fc4-44e2-868f-003043394609)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq" podUID="7b42415c-9fc4-44e2-868f-003043394609"
Jul 17 22:23:01.847838 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:01.847807    2565 scope.go:117] "RemoveContainer" containerID="15c0b79c530f246d3effb7e3f0c4833633db17a348ea45ef0c995749ff56a93f"
Jul 17 22:23:01.848328 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:23:01.848038    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq_odh-ai-gateway-infra(7b42415c-9fc4-44e2-868f-003043394609)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq" podUID="7b42415c-9fc4-44e2-868f-003043394609"
Jul 17 22:23:04.431609 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:04.431569    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j"]
Jul 17 22:23:04.432006 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:04.431996    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9401138f-6f70-4608-8873-84ab87ca77eb" containerName="revoke-keys"
Jul 17 22:23:04.432049 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:04.432010    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9401138f-6f70-4608-8873-84ab87ca77eb" containerName="revoke-keys"
Jul 17 22:23:04.432049 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:04.432030    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdedc97a-2a46-422c-bc72-1031ebee2871" containerName="revoke-keys"
Jul 17 22:23:04.432049 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:04.432038    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdedc97a-2a46-422c-bc72-1031ebee2871" containerName="revoke-keys"
Jul 17 22:23:04.432211 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:04.432138    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdedc97a-2a46-422c-bc72-1031ebee2871" containerName="revoke-keys"
Jul 17 22:23:04.436545 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:04.436524    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j"
Jul 17 22:23:04.442748 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:04.442705    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j"]
Jul 17 22:23:04.535981 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:04.535932    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j\" (UID: \"5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j"
Jul 17 22:23:04.636924 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:04.636884    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j\" (UID: \"5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j"
Jul 17 22:23:04.637612 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:04.637582    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j\" (UID: \"5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j"
Jul 17 22:23:04.749675 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:04.749639    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j"
Jul 17 22:23:04.895425 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:04.895399    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j"]
Jul 17 22:23:04.897632 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:23:04.897601    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2f94d8_3cf6_4d51_a4d8_1b64d40a71c8.slice/crio-32366c51f3874ddc55d6265e3ede878b9ba3f6a159f7811620c6f97ca01e0102 WatchSource:0}: Error finding container 32366c51f3874ddc55d6265e3ede878b9ba3f6a159f7811620c6f97ca01e0102: Status 404 returned error can't find the container with id 32366c51f3874ddc55d6265e3ede878b9ba3f6a159f7811620c6f97ca01e0102
Jul 17 22:23:05.869360 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:05.869324    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8" containerID="365f11ff61d2f72ec4dd2660effc337e74578089d2158c40d6cab24299b01c85" exitCode=7
Jul 17 22:23:05.869789 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:05.869421    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j" event={"ID":"5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8","Type":"ContainerDied","Data":"365f11ff61d2f72ec4dd2660effc337e74578089d2158c40d6cab24299b01c85"}
Jul 17 22:23:05.869789 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:05.869486    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j" event={"ID":"5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8","Type":"ContainerStarted","Data":"32366c51f3874ddc55d6265e3ede878b9ba3f6a159f7811620c6f97ca01e0102"}
Jul 17 22:23:05.869789 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:05.869599    2565 scope.go:117] "RemoveContainer" containerID="365f11ff61d2f72ec4dd2660effc337e74578089d2158c40d6cab24299b01c85"
Jul 17 22:23:06.876492 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:06.876457    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8" containerID="1db17af77c6fb7765dbae9219d25b07bf7d7da4a048a1eec27f2b6d6a75823e3" exitCode=7
Jul 17 22:23:06.876967 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:06.876554    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j" event={"ID":"5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8","Type":"ContainerDied","Data":"1db17af77c6fb7765dbae9219d25b07bf7d7da4a048a1eec27f2b6d6a75823e3"}
Jul 17 22:23:06.876967 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:06.876628    2565 scope.go:117] "RemoveContainer" containerID="365f11ff61d2f72ec4dd2660effc337e74578089d2158c40d6cab24299b01c85"
Jul 17 22:23:06.876967 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:06.876722    2565 scope.go:117] "RemoveContainer" containerID="1db17af77c6fb7765dbae9219d25b07bf7d7da4a048a1eec27f2b6d6a75823e3"
Jul 17 22:23:06.877128 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:23:06.876968    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j_odh-ai-gateway-infra(5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j" podUID="5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8"
Jul 17 22:23:07.883090 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:07.883045    2565 scope.go:117] "RemoveContainer" containerID="1db17af77c6fb7765dbae9219d25b07bf7d7da4a048a1eec27f2b6d6a75823e3"
Jul 17 22:23:07.883554 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:23:07.883349    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j_odh-ai-gateway-infra(5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j" podUID="5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8"
Jul 17 22:23:15.177408 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:15.177365    2565 scope.go:117] "RemoveContainer" containerID="15c0b79c530f246d3effb7e3f0c4833633db17a348ea45ef0c995749ff56a93f"
Jul 17 22:23:15.929437 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:15.929400    2565 generic.go:356] "Generic (PLEG): container finished" podID="7b42415c-9fc4-44e2-868f-003043394609" containerID="2b4ab9ae248d90bc8f0b73ac21ad7a638aaeee4fc5b7e7d78a63eab5ff226acb" exitCode=6
Jul 17 22:23:15.929643 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:15.929482    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq" event={"ID":"7b42415c-9fc4-44e2-868f-003043394609","Type":"ContainerDied","Data":"2b4ab9ae248d90bc8f0b73ac21ad7a638aaeee4fc5b7e7d78a63eab5ff226acb"}
Jul 17 22:23:15.929643 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:15.929544    2565 scope.go:117] "RemoveContainer" containerID="15c0b79c530f246d3effb7e3f0c4833633db17a348ea45ef0c995749ff56a93f"
Jul 17 22:23:15.929869 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:15.929853    2565 scope.go:117] "RemoveContainer" containerID="2b4ab9ae248d90bc8f0b73ac21ad7a638aaeee4fc5b7e7d78a63eab5ff226acb"
Jul 17 22:23:15.930161 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:23:15.930136    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq_odh-ai-gateway-infra(7b42415c-9fc4-44e2-868f-003043394609)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq" podUID="7b42415c-9fc4-44e2-868f-003043394609"
Jul 17 22:23:15.957190 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:15.957158    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn"]
Jul 17 22:23:15.961725 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:15.961701    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn"
Jul 17 22:23:15.967730 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:15.967677    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn"]
Jul 17 22:23:16.043047 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:16.043010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn\" (UID: \"7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn"
Jul 17 22:23:16.144316 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:16.144203    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn\" (UID: \"7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn"
Jul 17 22:23:16.144852 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:16.144829    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn\" (UID: \"7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn"
Jul 17 22:23:16.274387 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:16.274350    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn"
Jul 17 22:23:16.414631 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:16.414552    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn"]
Jul 17 22:23:16.417924 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:23:16.417889    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3b8329_d5e2_4fa2_8c24_bd326fd1ef34.slice/crio-9e05edcd3f2bc59beea7c02ac6fe0f1a9350a184d82916362554c7ce670ecb5b WatchSource:0}: Error finding container 9e05edcd3f2bc59beea7c02ac6fe0f1a9350a184d82916362554c7ce670ecb5b: Status 404 returned error can't find the container with id 9e05edcd3f2bc59beea7c02ac6fe0f1a9350a184d82916362554c7ce670ecb5b
Jul 17 22:23:16.938018 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:16.937984    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn" event={"ID":"7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34","Type":"ContainerStarted","Data":"838c9b5652a1863b772974981fde18956d3a020b42cf640b2aaa9cdc1384406d"}
Jul 17 22:23:16.938241 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:16.938036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn" event={"ID":"7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34","Type":"ContainerStarted","Data":"9e05edcd3f2bc59beea7c02ac6fe0f1a9350a184d82916362554c7ce670ecb5b"}
Jul 17 22:23:16.955480 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:16.955430    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn" podStartSLOduration=1.955412457 podStartE2EDuration="1.955412457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:23:16.953393609 +0000 UTC m=+2260.342410279" watchObservedRunningTime="2026-07-17 22:23:16.955412457 +0000 UTC m=+2260.344429086"
Jul 17 22:23:16.961651 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:16.961618    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq"]
Jul 17 22:23:17.092964 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:17.092937    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq"
Jul 17 22:23:17.155244 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:17.155213    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b42415c-9fc4-44e2-868f-003043394609-maas-api-service-ca\") pod \"7b42415c-9fc4-44e2-868f-003043394609\" (UID: \"7b42415c-9fc4-44e2-868f-003043394609\") "
Jul 17 22:23:17.155591 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:17.155557    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b42415c-9fc4-44e2-868f-003043394609-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b42415c-9fc4-44e2-868f-003043394609" (UID: "7b42415c-9fc4-44e2-868f-003043394609"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:23:17.256657 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:17.256606    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b42415c-9fc4-44e2-868f-003043394609-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:23:17.944306 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:17.944271    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq"
Jul 17 22:23:17.944306 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:17.944294    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq" event={"ID":"7b42415c-9fc4-44e2-868f-003043394609","Type":"ContainerDied","Data":"1fd9797e51f1d4337c0414831fa6974553abacb0d872861595f81605a122f096"}
Jul 17 22:23:17.944851 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:17.944358    2565 scope.go:117] "RemoveContainer" containerID="2b4ab9ae248d90bc8f0b73ac21ad7a638aaeee4fc5b7e7d78a63eab5ff226acb"
Jul 17 22:23:17.946271 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:17.946238    2565 generic.go:356] "Generic (PLEG): container finished" podID="7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34" containerID="838c9b5652a1863b772974981fde18956d3a020b42cf640b2aaa9cdc1384406d" exitCode=6
Jul 17 22:23:17.946398 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:17.946297    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn" event={"ID":"7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34","Type":"ContainerDied","Data":"838c9b5652a1863b772974981fde18956d3a020b42cf640b2aaa9cdc1384406d"}
Jul 17 22:23:17.946617 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:17.946600    2565 scope.go:117] "RemoveContainer" containerID="838c9b5652a1863b772974981fde18956d3a020b42cf640b2aaa9cdc1384406d"
Jul 17 22:23:17.960706 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:17.960683    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq"]
Jul 17 22:23:17.966005 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:17.965974    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j7ztq"]
Jul 17 22:23:18.955788 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:18.955676    2565 generic.go:356] "Generic (PLEG): container finished" podID="7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34" containerID="7fdb252503a6f12619e5121e14ee45e09ae2c018e2761f5ce04a8f799b33fbd5" exitCode=6
Jul 17 22:23:18.955788 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:18.955761    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn" event={"ID":"7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34","Type":"ContainerDied","Data":"7fdb252503a6f12619e5121e14ee45e09ae2c018e2761f5ce04a8f799b33fbd5"}
Jul 17 22:23:18.956309 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:18.955873    2565 scope.go:117] "RemoveContainer" containerID="838c9b5652a1863b772974981fde18956d3a020b42cf640b2aaa9cdc1384406d"
Jul 17 22:23:18.956309 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:18.956180    2565 scope.go:117] "RemoveContainer" containerID="7fdb252503a6f12619e5121e14ee45e09ae2c018e2761f5ce04a8f799b33fbd5"
Jul 17 22:23:18.956424 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:23:18.956400    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn_odh-ai-gateway-infra(7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn" podUID="7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34"
Jul 17 22:23:19.002725 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:19.002694    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:19.057160 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:19.057127    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:19.182711 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:19.182675    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b42415c-9fc4-44e2-868f-003043394609" path="/var/lib/kubelet/pods/7b42415c-9fc4-44e2-868f-003043394609/volumes"
Jul 17 22:23:19.962594 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:19.962568    2565 scope.go:117] "RemoveContainer" containerID="7fdb252503a6f12619e5121e14ee45e09ae2c018e2761f5ce04a8f799b33fbd5"
Jul 17 22:23:19.962999 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:23:19.962801    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn_odh-ai-gateway-infra(7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn" podUID="7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34"
Jul 17 22:23:20.177035 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:20.177002    2565 scope.go:117] "RemoveContainer" containerID="1db17af77c6fb7765dbae9219d25b07bf7d7da4a048a1eec27f2b6d6a75823e3"
Jul 17 22:23:20.969606 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:20.969570    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8" containerID="bbf65b6053a33d57ca14e5319b94304246527b1959fbd22d5e9345f9b85c1c16" exitCode=7
Jul 17 22:23:20.970018 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:20.969642    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j" event={"ID":"5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8","Type":"ContainerDied","Data":"bbf65b6053a33d57ca14e5319b94304246527b1959fbd22d5e9345f9b85c1c16"}
Jul 17 22:23:20.970018 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:20.969719    2565 scope.go:117] "RemoveContainer" containerID="1db17af77c6fb7765dbae9219d25b07bf7d7da4a048a1eec27f2b6d6a75823e3"
Jul 17 22:23:20.970180 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:20.970125    2565 scope.go:117] "RemoveContainer" containerID="bbf65b6053a33d57ca14e5319b94304246527b1959fbd22d5e9345f9b85c1c16"
Jul 17 22:23:20.970403 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:23:20.970374    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j_odh-ai-gateway-infra(5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j" podUID="5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8"
Jul 17 22:23:21.994608 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:21.994568    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j"]
Jul 17 22:23:22.135986 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:22.135958    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j"
Jul 17 22:23:22.204679 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:22.204639    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8-maas-api-service-ca\") pod \"5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8\" (UID: \"5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8\") "
Jul 17 22:23:22.205042 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:22.205020    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8" (UID: "5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:23:22.305950 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:22.305914    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:23:22.981993 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:22.981958    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j"
Jul 17 22:23:22.982217 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:22.982006    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j" event={"ID":"5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8","Type":"ContainerDied","Data":"32366c51f3874ddc55d6265e3ede878b9ba3f6a159f7811620c6f97ca01e0102"}
Jul 17 22:23:22.982217 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:22.982089    2565 scope.go:117] "RemoveContainer" containerID="bbf65b6053a33d57ca14e5319b94304246527b1959fbd22d5e9345f9b85c1c16"
Jul 17 22:23:23.009085 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:23.009036    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j"]
Jul 17 22:23:23.021817 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:23.021780    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-sgp4j"]
Jul 17 22:23:23.181868 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:23.181830    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8" path="/var/lib/kubelet/pods/5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8/volumes"
Jul 17 22:23:24.372296 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:24.372258    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"]
Jul 17 22:23:24.372683 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:24.372501    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts" podUID="edcfbc0e-024d-41b6-a49d-f759bd3e53b6" containerName="istio-proxy" containerID="cri-o://94414cd074013d0ae5c685dca83e4ce82d672b418fc90c1a592b5d4b18453038" gracePeriod=30
Jul 17 22:23:24.396583 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:24.396548    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:24.505829 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:24.505766    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:27.241420 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:27.241384    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:23:29.633568 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.633498    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:23:29.777142 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.777103    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istiod-ca-cert\") pod \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") "
Jul 17 22:23:29.777347 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.777203    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ckds\" (UniqueName: \"kubernetes.io/projected/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-kube-api-access-2ckds\") pod \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") "
Jul 17 22:23:29.777347 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.777234    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-workload-certs\") pod \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") "
Jul 17 22:23:29.777347 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.777256    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-envoy\") pod \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") "
Jul 17 22:23:29.777347 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.777286    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-token\") pod \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") "
Jul 17 22:23:29.777347 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.777307    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-workload-socket\") pod \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") "
Jul 17 22:23:29.777347 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.777331    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-data\") pod \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") "
Jul 17 22:23:29.777659 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.777360    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-podinfo\") pod \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") "
Jul 17 22:23:29.777659 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.777397    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-credential-socket\") pod \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\" (UID: \"edcfbc0e-024d-41b6-a49d-f759bd3e53b6\") "
Jul 17 22:23:29.777659 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.777524    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "edcfbc0e-024d-41b6-a49d-f759bd3e53b6" (UID: "edcfbc0e-024d-41b6-a49d-f759bd3e53b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:23:29.777659 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.777544    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "edcfbc0e-024d-41b6-a49d-f759bd3e53b6" (UID: "edcfbc0e-024d-41b6-a49d-f759bd3e53b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:23:29.777659 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.777638    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "edcfbc0e-024d-41b6-a49d-f759bd3e53b6" (UID: "edcfbc0e-024d-41b6-a49d-f759bd3e53b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:23:29.777900 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.777729    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istiod-ca-cert\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:23:29.777900 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.777748    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-workload-certs\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:23:29.777900 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.777761    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-workload-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:23:29.777900 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.777840    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "edcfbc0e-024d-41b6-a49d-f759bd3e53b6" (UID: "edcfbc0e-024d-41b6-a49d-f759bd3e53b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:23:29.778458 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.778430    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "edcfbc0e-024d-41b6-a49d-f759bd3e53b6" (UID: "edcfbc0e-024d-41b6-a49d-f759bd3e53b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:23:29.779661 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.779633    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "edcfbc0e-024d-41b6-a49d-f759bd3e53b6" (UID: "edcfbc0e-024d-41b6-a49d-f759bd3e53b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:23:29.779771 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.779687    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-kube-api-access-2ckds" (OuterVolumeSpecName: "kube-api-access-2ckds") pod "edcfbc0e-024d-41b6-a49d-f759bd3e53b6" (UID: "edcfbc0e-024d-41b6-a49d-f759bd3e53b6"). InnerVolumeSpecName "kube-api-access-2ckds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:23:29.779771 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.779695    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "edcfbc0e-024d-41b6-a49d-f759bd3e53b6" (UID: "edcfbc0e-024d-41b6-a49d-f759bd3e53b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:23:29.779771 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.779759    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "edcfbc0e-024d-41b6-a49d-f759bd3e53b6" (UID: "edcfbc0e-024d-41b6-a49d-f759bd3e53b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:23:29.878441 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.878399    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ckds\" (UniqueName: \"kubernetes.io/projected/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-kube-api-access-2ckds\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:23:29.878441 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.878432    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-envoy\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:23:29.878441 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.878442    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-token\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:23:29.878441 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.878450    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-data\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:23:29.878706 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.878458    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-istio-podinfo\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:23:29.878706 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:29.878466    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edcfbc0e-024d-41b6-a49d-f759bd3e53b6-credential-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:23:30.021543 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:30.021500    2565 generic.go:356] "Generic (PLEG): container finished" podID="edcfbc0e-024d-41b6-a49d-f759bd3e53b6" containerID="94414cd074013d0ae5c685dca83e4ce82d672b418fc90c1a592b5d4b18453038" exitCode=0
Jul 17 22:23:30.021704 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:30.021549    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"
Jul 17 22:23:30.021704 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:30.021585    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts" event={"ID":"edcfbc0e-024d-41b6-a49d-f759bd3e53b6","Type":"ContainerDied","Data":"94414cd074013d0ae5c685dca83e4ce82d672b418fc90c1a592b5d4b18453038"}
Jul 17 22:23:30.021704 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:30.021660    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts" event={"ID":"edcfbc0e-024d-41b6-a49d-f759bd3e53b6","Type":"ContainerDied","Data":"44f60e2c7af1a4fa25197030eeaceb89b7ac10af7f09b298da7b34edb6c4ab12"}
Jul 17 22:23:30.021704 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:30.021686    2565 scope.go:117] "RemoveContainer" containerID="94414cd074013d0ae5c685dca83e4ce82d672b418fc90c1a592b5d4b18453038"
Jul 17 22:23:30.027233 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:30.027208    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:30.031245 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:30.031228    2565 scope.go:117] "RemoveContainer" containerID="30a35eea2ce7ca4bb75d11357bf6392ec8638133d63b3e2aaebf86ebc507247d"
Jul 17 22:23:30.039545 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:30.039526    2565 scope.go:117] "RemoveContainer" containerID="94414cd074013d0ae5c685dca83e4ce82d672b418fc90c1a592b5d4b18453038"
Jul 17 22:23:30.039815 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:23:30.039795    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94414cd074013d0ae5c685dca83e4ce82d672b418fc90c1a592b5d4b18453038\": container with ID starting with 94414cd074013d0ae5c685dca83e4ce82d672b418fc90c1a592b5d4b18453038 not found: ID does not exist" containerID="94414cd074013d0ae5c685dca83e4ce82d672b418fc90c1a592b5d4b18453038"
Jul 17 22:23:30.039860 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:30.039826    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94414cd074013d0ae5c685dca83e4ce82d672b418fc90c1a592b5d4b18453038"} err="failed to get container status \"94414cd074013d0ae5c685dca83e4ce82d672b418fc90c1a592b5d4b18453038\": rpc error: code = NotFound desc = could not find container \"94414cd074013d0ae5c685dca83e4ce82d672b418fc90c1a592b5d4b18453038\": container with ID starting with 94414cd074013d0ae5c685dca83e4ce82d672b418fc90c1a592b5d4b18453038 not found: ID does not exist"
Jul 17 22:23:30.039860 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:30.039846    2565 scope.go:117] "RemoveContainer" containerID="30a35eea2ce7ca4bb75d11357bf6392ec8638133d63b3e2aaebf86ebc507247d"
Jul 17 22:23:30.040113 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:23:30.040092    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a35eea2ce7ca4bb75d11357bf6392ec8638133d63b3e2aaebf86ebc507247d\": container with ID starting with 30a35eea2ce7ca4bb75d11357bf6392ec8638133d63b3e2aaebf86ebc507247d not found: ID does not exist" containerID="30a35eea2ce7ca4bb75d11357bf6392ec8638133d63b3e2aaebf86ebc507247d"
Jul 17 22:23:30.040151 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:30.040121    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a35eea2ce7ca4bb75d11357bf6392ec8638133d63b3e2aaebf86ebc507247d"} err="failed to get container status \"30a35eea2ce7ca4bb75d11357bf6392ec8638133d63b3e2aaebf86ebc507247d\": rpc error: code = NotFound desc = could not find container \"30a35eea2ce7ca4bb75d11357bf6392ec8638133d63b3e2aaebf86ebc507247d\": container with ID starting with 30a35eea2ce7ca4bb75d11357bf6392ec8638133d63b3e2aaebf86ebc507247d not found: ID does not exist"
Jul 17 22:23:30.046667 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:30.046526    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"]
Jul 17 22:23:30.048756 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:30.048729    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-45495ac6-openshift-default-6fbcf4d469-9kqts"]
Jul 17 22:23:30.715780 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:30.715750    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:31.181886 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:31.181853    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edcfbc0e-024d-41b6-a49d-f759bd3e53b6" path="/var/lib/kubelet/pods/edcfbc0e-024d-41b6-a49d-f759bd3e53b6/volumes"
Jul 17 22:23:35.177254 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:35.177223    2565 scope.go:117] "RemoveContainer" containerID="7fdb252503a6f12619e5121e14ee45e09ae2c018e2761f5ce04a8f799b33fbd5"
Jul 17 22:23:36.053824 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.053731    2565 generic.go:356] "Generic (PLEG): container finished" podID="7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34" containerID="e5163541be04ec096f72b03224bf151a9d6c808b4aff16ea2ebbb6fe293f0fdd" exitCode=6
Jul 17 22:23:36.053824 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.053801    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn" event={"ID":"7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34","Type":"ContainerDied","Data":"e5163541be04ec096f72b03224bf151a9d6c808b4aff16ea2ebbb6fe293f0fdd"}
Jul 17 22:23:36.054055 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.053863    2565 scope.go:117] "RemoveContainer" containerID="7fdb252503a6f12619e5121e14ee45e09ae2c018e2761f5ce04a8f799b33fbd5"
Jul 17 22:23:36.054195 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.054177    2565 scope.go:117] "RemoveContainer" containerID="e5163541be04ec096f72b03224bf151a9d6c808b4aff16ea2ebbb6fe293f0fdd"
Jul 17 22:23:36.054429 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:23:36.054404    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn_odh-ai-gateway-infra(7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn" podUID="7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34"
Jul 17 22:23:36.545794 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.545760    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:36.647091 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.647014    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:36.784771 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.784736    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"]
Jul 17 22:23:36.785156 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.785137    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b42415c-9fc4-44e2-868f-003043394609" containerName="revoke-keys"
Jul 17 22:23:36.785156 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.785154    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b42415c-9fc4-44e2-868f-003043394609" containerName="revoke-keys"
Jul 17 22:23:36.785352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.785164    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8" containerName="revoke-keys"
Jul 17 22:23:36.785352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.785170    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8" containerName="revoke-keys"
Jul 17 22:23:36.785352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.785185    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edcfbc0e-024d-41b6-a49d-f759bd3e53b6" containerName="istio-proxy"
Jul 17 22:23:36.785352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.785195    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="edcfbc0e-024d-41b6-a49d-f759bd3e53b6" containerName="istio-proxy"
Jul 17 22:23:36.785352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.785216    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b42415c-9fc4-44e2-868f-003043394609" containerName="revoke-keys"
Jul 17 22:23:36.785352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.785221    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b42415c-9fc4-44e2-868f-003043394609" containerName="revoke-keys"
Jul 17 22:23:36.785352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.785227    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8" containerName="revoke-keys"
Jul 17 22:23:36.785352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.785233    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8" containerName="revoke-keys"
Jul 17 22:23:36.785352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.785295    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8" containerName="revoke-keys"
Jul 17 22:23:36.785352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.785303    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="edcfbc0e-024d-41b6-a49d-f759bd3e53b6" containerName="istio-proxy"
Jul 17 22:23:36.785352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.785310    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8" containerName="revoke-keys"
Jul 17 22:23:36.785352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.785315    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b42415c-9fc4-44e2-868f-003043394609" containerName="revoke-keys"
Jul 17 22:23:36.785352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.785320    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b42415c-9fc4-44e2-868f-003043394609" containerName="revoke-keys"
Jul 17 22:23:36.785352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.785327    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="edcfbc0e-024d-41b6-a49d-f759bd3e53b6" containerName="istio-proxy"
Jul 17 22:23:36.789680 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.789658    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:36.792032 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.792015    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ded85f30-openshift-default-dockercfg-vhh6l\""
Jul 17 22:23:36.800495 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.800471    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"]
Jul 17 22:23:36.817701 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.817653    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:36.946817 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.946778    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-podinfo\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:36.947037 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.946842    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-credential-socket\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:36.947037 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.946922    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-envoy\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:36.947037 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.946979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-workload-certs\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:36.947037 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.947022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-workload-socket\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:36.947299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.947045    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-token\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:36.947299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.947138    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istiod-ca-cert\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:36.947299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.947167    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8lls\" (UniqueName: \"kubernetes.io/projected/94bb42a4-ced6-4a58-9b65-a360794d4d0b-kube-api-access-l8lls\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:36.947299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:36.947221    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-data\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.047922 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.047823    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istiod-ca-cert\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.047922 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.047870    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8lls\" (UniqueName: \"kubernetes.io/projected/94bb42a4-ced6-4a58-9b65-a360794d4d0b-kube-api-access-l8lls\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.047922 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.047890    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-data\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.047922 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.047914    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-podinfo\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.048266 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.047959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-credential-socket\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.048266 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.047984    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-envoy\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.048266 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.048022    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-workload-certs\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.048266 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.048060    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-workload-socket\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.048266 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.048104    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-token\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.048528 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.048422    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-data\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.048528 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.048487    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-credential-socket\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.048637 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.048585    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-workload-certs\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.048696 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.048639    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-workload-socket\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.048985 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.048959    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istiod-ca-cert\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.050685 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.050656    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-envoy\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.050818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.050761    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-podinfo\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.059860 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.059823    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-token\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.060052 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.060026    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8lls\" (UniqueName: \"kubernetes.io/projected/94bb42a4-ced6-4a58-9b65-a360794d4d0b-kube-api-access-l8lls\") pod \"e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") " pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.082480 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.082444    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn"]
Jul 17 22:23:37.103808 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.103780    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ded85f30-openshift-default-dockercfg-vhh6l\""
Jul 17 22:23:37.111969 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.111935    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:37.234704 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.234677    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn"
Jul 17 22:23:37.277402 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.277373    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"]
Jul 17 22:23:37.278920 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:23:37.278892    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bb42a4_ced6_4a58_9b65_a360794d4d0b.slice/crio-1d13474ef99ba04b36c6a929ac0ada5004aadc5953d431c47f29b103374c4d47 WatchSource:0}: Error finding container 1d13474ef99ba04b36c6a929ac0ada5004aadc5953d431c47f29b103374c4d47: Status 404 returned error can't find the container with id 1d13474ef99ba04b36c6a929ac0ada5004aadc5953d431c47f29b103374c4d47
Jul 17 22:23:37.281199 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.281167    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:23:37.281269 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.281231    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:23:37.281269 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.281260    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:23:37.350173 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.350139    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34-maas-api-service-ca\") pod \"7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34\" (UID: \"7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34\") "
Jul 17 22:23:37.350479 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.350457    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34" (UID: "7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:23:37.451102 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.451037    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:23:37.963668 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:37.963636    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:38.068402 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:38.068368    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg" event={"ID":"94bb42a4-ced6-4a58-9b65-a360794d4d0b","Type":"ContainerStarted","Data":"43776e0736e8502f70f789faac0b5749ca94c57026acf97936ae4f0e8a17351a"}
Jul 17 22:23:38.068591 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:38.068434    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg" event={"ID":"94bb42a4-ced6-4a58-9b65-a360794d4d0b","Type":"ContainerStarted","Data":"1d13474ef99ba04b36c6a929ac0ada5004aadc5953d431c47f29b103374c4d47"}
Jul 17 22:23:38.069635 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:38.069612    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn" event={"ID":"7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34","Type":"ContainerDied","Data":"9e05edcd3f2bc59beea7c02ac6fe0f1a9350a184d82916362554c7ce670ecb5b"}
Jul 17 22:23:38.069734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:38.069648    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn"
Jul 17 22:23:38.069734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:38.069665    2565 scope.go:117] "RemoveContainer" containerID="e5163541be04ec096f72b03224bf151a9d6c808b4aff16ea2ebbb6fe293f0fdd"
Jul 17 22:23:38.089215 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:38.089182    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:38.104438 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:38.104383    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg" podStartSLOduration=2.104368947 podStartE2EDuration="2.104368947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:23:38.103569043 +0000 UTC m=+2281.492585714" watchObservedRunningTime="2026-07-17 22:23:38.104368947 +0000 UTC m=+2281.493385597"
Jul 17 22:23:38.112470 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:38.112427    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:38.117660 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:38.117623    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:38.122131 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:38.122106    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn"]
Jul 17 22:23:38.127464 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:38.127436    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-ndqgn"]
Jul 17 22:23:38.189966 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:38.189929    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:38.298557 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:38.298520    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:38.694557 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:38.694474    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:38.797420 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:38.797379    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:39.075798 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:39.075766    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:39.076842 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:39.076823    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:39.181878 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:39.181836    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34" path="/var/lib/kubelet/pods/7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34/volumes"
Jul 17 22:23:41.044786 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:41.044740    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:23:42.345141 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345100    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv"]
Jul 17 22:23:42.345522 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345500    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34" containerName="revoke-keys"
Jul 17 22:23:42.345522 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345514    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34" containerName="revoke-keys"
Jul 17 22:23:42.345522 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345523    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8" containerName="revoke-keys"
Jul 17 22:23:42.345618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345528    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8" containerName="revoke-keys"
Jul 17 22:23:42.345618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345536    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34" containerName="revoke-keys"
Jul 17 22:23:42.345618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345541    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34" containerName="revoke-keys"
Jul 17 22:23:42.345618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345552    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b42415c-9fc4-44e2-868f-003043394609" containerName="revoke-keys"
Jul 17 22:23:42.345618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345558    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b42415c-9fc4-44e2-868f-003043394609" containerName="revoke-keys"
Jul 17 22:23:42.345618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345573    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edcfbc0e-024d-41b6-a49d-f759bd3e53b6" containerName="istio-proxy"
Jul 17 22:23:42.345618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345580    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="edcfbc0e-024d-41b6-a49d-f759bd3e53b6" containerName="istio-proxy"
Jul 17 22:23:42.345618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345589    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34" containerName="revoke-keys"
Jul 17 22:23:42.345618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345594    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34" containerName="revoke-keys"
Jul 17 22:23:42.345887 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345663    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34" containerName="revoke-keys"
Jul 17 22:23:42.345887 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345673    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34" containerName="revoke-keys"
Jul 17 22:23:42.345887 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345681    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d2f94d8-3cf6-4d51-a4d8-1b64d40a71c8" containerName="revoke-keys"
Jul 17 22:23:42.345887 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.345689    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b42415c-9fc4-44e2-868f-003043394609" containerName="revoke-keys"
Jul 17 22:23:42.348784 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.348764    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv"
Jul 17 22:23:42.351162 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.351134    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:23:42.351162 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.351155    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:23:42.356842 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.356813    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv"]
Jul 17 22:23:42.396631 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.396584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv\" (UID: \"7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv"
Jul 17 22:23:42.497980 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.497936    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv\" (UID: \"7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv"
Jul 17 22:23:42.498577 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.498558    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv\" (UID: \"7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv"
Jul 17 22:23:42.662976 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.662886    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv"
Jul 17 22:23:42.833874 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:42.833689    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv"]
Jul 17 22:23:42.836595 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:23:42.836564    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb1add7_fe38_42d5_a3ba_c1e4b934a9ab.slice/crio-86555ca75d50422986b23c24a927c42f1ee6693b3f6cf0080e90cbe0c411fd89 WatchSource:0}: Error finding container 86555ca75d50422986b23c24a927c42f1ee6693b3f6cf0080e90cbe0c411fd89: Status 404 returned error can't find the container with id 86555ca75d50422986b23c24a927c42f1ee6693b3f6cf0080e90cbe0c411fd89
Jul 17 22:23:43.095841 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:43.095802    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv" event={"ID":"7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab","Type":"ContainerStarted","Data":"86555ca75d50422986b23c24a927c42f1ee6693b3f6cf0080e90cbe0c411fd89"}
Jul 17 22:23:44.101615 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:44.101520    2565 generic.go:356] "Generic (PLEG): container finished" podID="7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab" containerID="0ab249edba5c11be2c586edffe4a37e7b03676e42d9e7b9956f5f5c56a75b05a" exitCode=7
Jul 17 22:23:44.101966 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:44.101606    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv" event={"ID":"7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab","Type":"ContainerDied","Data":"0ab249edba5c11be2c586edffe4a37e7b03676e42d9e7b9956f5f5c56a75b05a"}
Jul 17 22:23:44.101966 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:44.101748    2565 scope.go:117] "RemoveContainer" containerID="0ab249edba5c11be2c586edffe4a37e7b03676e42d9e7b9956f5f5c56a75b05a"
Jul 17 22:23:45.108572 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:45.108503    2565 generic.go:356] "Generic (PLEG): container finished" podID="7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab" containerID="6954f4f10dd4a8d5e69b835063d048ce485493f79e6c6b31188ce508a8e96eaf" exitCode=7
Jul 17 22:23:45.108572 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:45.108570    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv" event={"ID":"7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab","Type":"ContainerDied","Data":"6954f4f10dd4a8d5e69b835063d048ce485493f79e6c6b31188ce508a8e96eaf"}
Jul 17 22:23:45.109082 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:45.108616    2565 scope.go:117] "RemoveContainer" containerID="0ab249edba5c11be2c586edffe4a37e7b03676e42d9e7b9956f5f5c56a75b05a"
Jul 17 22:23:45.109082 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:45.108747    2565 scope.go:117] "RemoveContainer" containerID="6954f4f10dd4a8d5e69b835063d048ce485493f79e6c6b31188ce508a8e96eaf"
Jul 17 22:23:45.109082 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:23:45.108986    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv_odh-ai-gateway-infra(7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv" podUID="7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab"
Jul 17 22:23:45.780606 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:45.780475    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:45.879839 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:45.879802    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:46.116307 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:46.116232    2565 scope.go:117] "RemoveContainer" containerID="6954f4f10dd4a8d5e69b835063d048ce485493f79e6c6b31188ce508a8e96eaf"
Jul 17 22:23:46.116651 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:23:46.116510    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv_odh-ai-gateway-infra(7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv" podUID="7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab"
Jul 17 22:23:49.131736 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:49.131702    2565 generic.go:356] "Generic (PLEG): container finished" podID="94bb42a4-ced6-4a58-9b65-a360794d4d0b" containerID="43776e0736e8502f70f789faac0b5749ca94c57026acf97936ae4f0e8a17351a" exitCode=0
Jul 17 22:23:49.132228 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:49.131777    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg" event={"ID":"94bb42a4-ced6-4a58-9b65-a360794d4d0b","Type":"ContainerDied","Data":"43776e0736e8502f70f789faac0b5749ca94c57026acf97936ae4f0e8a17351a"}
Jul 17 22:23:49.132228 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:49.132193    2565 scope.go:117] "RemoveContainer" containerID="43776e0736e8502f70f789faac0b5749ca94c57026acf97936ae4f0e8a17351a"
Jul 17 22:23:49.133201 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:49.133171    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:23:49.133277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:49.133243    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:23:49.133317 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:49.133276    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:23:49.596083 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:49.596031    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:23:50.139277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:50.139235    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg" event={"ID":"94bb42a4-ced6-4a58-9b65-a360794d4d0b","Type":"ContainerStarted","Data":"4b1af1d9e5e7ef66dc101ae2224f63f1551f0b18575a57af2109f9a8f7caabfb"}
Jul 17 22:23:51.112992 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:51.112946    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:51.114280 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:51.114254    2565 patch_prober.go:28] interesting pod/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 17 22:23:51.114362 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:51.114308    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg" podUID="94bb42a4-ced6-4a58-9b65-a360794d4d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 17 22:23:52.112418 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:52.112381    2565 patch_prober.go:28] interesting pod/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 17 22:23:52.112839 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:52.112438    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg" podUID="94bb42a4-ced6-4a58-9b65-a360794d4d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 17 22:23:53.112449 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:53.112408    2565 patch_prober.go:28] interesting pod/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 17 22:23:53.112858 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:53.112470    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg" podUID="94bb42a4-ced6-4a58-9b65-a360794d4d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 17 22:23:54.076149 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:54.076103    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:54.124683 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:54.124650    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:54.158963 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:54.158934    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:23:54.477905 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:54.477820    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq"]
Jul 17 22:23:54.478334 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:54.478318    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3b8329-d5e2-4fa2-8c24-bd326fd1ef34" containerName="revoke-keys"
Jul 17 22:23:54.481956 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:54.481936    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq"
Jul 17 22:23:54.489374 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:54.489343    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq"]
Jul 17 22:23:54.609327 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:54.609291    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/063d1362-cbe3-4a75-b278-48f804a067b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq\" (UID: \"063d1362-cbe3-4a75-b278-48f804a067b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq"
Jul 17 22:23:54.710673 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:54.710631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/063d1362-cbe3-4a75-b278-48f804a067b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq\" (UID: \"063d1362-cbe3-4a75-b278-48f804a067b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq"
Jul 17 22:23:54.711307 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:54.711287    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/063d1362-cbe3-4a75-b278-48f804a067b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq\" (UID: \"063d1362-cbe3-4a75-b278-48f804a067b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq"
Jul 17 22:23:54.796219 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:54.796183    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq"
Jul 17 22:23:54.936530 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:54.936478    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq"]
Jul 17 22:23:54.938887 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:23:54.938856    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063d1362_cbe3_4a75_b278_48f804a067b5.slice/crio-7d10e96d9a408aa9623c159ac3da7d99e832798abc8f14717c9fe29c1aa7da95 WatchSource:0}: Error finding container 7d10e96d9a408aa9623c159ac3da7d99e832798abc8f14717c9fe29c1aa7da95: Status 404 returned error can't find the container with id 7d10e96d9a408aa9623c159ac3da7d99e832798abc8f14717c9fe29c1aa7da95
Jul 17 22:23:55.164728 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:55.164651    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq" event={"ID":"063d1362-cbe3-4a75-b278-48f804a067b5","Type":"ContainerStarted","Data":"7d10e96d9a408aa9623c159ac3da7d99e832798abc8f14717c9fe29c1aa7da95"}
Jul 17 22:23:56.170628 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:56.170590    2565 generic.go:356] "Generic (PLEG): container finished" podID="063d1362-cbe3-4a75-b278-48f804a067b5" containerID="ce5620b81ed6993c8394fa49179a693c5828581dc6fe9879c7344bd41d04747e" exitCode=6
Jul 17 22:23:56.171025 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:56.170678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq" event={"ID":"063d1362-cbe3-4a75-b278-48f804a067b5","Type":"ContainerDied","Data":"ce5620b81ed6993c8394fa49179a693c5828581dc6fe9879c7344bd41d04747e"}
Jul 17 22:23:56.171025 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:56.170834    2565 scope.go:117] "RemoveContainer" containerID="ce5620b81ed6993c8394fa49179a693c5828581dc6fe9879c7344bd41d04747e"
Jul 17 22:23:57.177970 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:57.177877    2565 generic.go:356] "Generic (PLEG): container finished" podID="063d1362-cbe3-4a75-b278-48f804a067b5" containerID="575bc341001116e8ce68154e800f151a9d6dc127a4265e54b9dd6f26574ee307" exitCode=6
Jul 17 22:23:57.178766 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:57.178750    2565 scope.go:117] "RemoveContainer" containerID="575bc341001116e8ce68154e800f151a9d6dc127a4265e54b9dd6f26574ee307"
Jul 17 22:23:57.178974 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:23:57.178956    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq_odh-ai-gateway-infra(063d1362-cbe3-4a75-b278-48f804a067b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq" podUID="063d1362-cbe3-4a75-b278-48f804a067b5"
Jul 17 22:23:57.181945 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:57.181918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq" event={"ID":"063d1362-cbe3-4a75-b278-48f804a067b5","Type":"ContainerDied","Data":"575bc341001116e8ce68154e800f151a9d6dc127a4265e54b9dd6f26574ee307"}
Jul 17 22:23:57.182095 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:57.181978    2565 scope.go:117] "RemoveContainer" containerID="ce5620b81ed6993c8394fa49179a693c5828581dc6fe9879c7344bd41d04747e"
Jul 17 22:23:57.803584 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:57.803548    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg"]
Jul 17 22:23:57.808266 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:57.808250    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg"
Jul 17 22:23:57.813655 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:57.813622    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg"]
Jul 17 22:23:57.945130 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:57.945089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4f00e5-2350-4741-ab28-31e9a16ff3a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg\" (UID: \"ed4f00e5-2350-4741-ab28-31e9a16ff3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg"
Jul 17 22:23:58.046449 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:58.046408    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4f00e5-2350-4741-ab28-31e9a16ff3a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg\" (UID: \"ed4f00e5-2350-4741-ab28-31e9a16ff3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg"
Jul 17 22:23:58.047111 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:58.047057    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4f00e5-2350-4741-ab28-31e9a16ff3a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg\" (UID: \"ed4f00e5-2350-4741-ab28-31e9a16ff3a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg"
Jul 17 22:23:58.122316 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:58.122233    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg"
Jul 17 22:23:58.176713 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:58.176687    2565 scope.go:117] "RemoveContainer" containerID="6954f4f10dd4a8d5e69b835063d048ce485493f79e6c6b31188ce508a8e96eaf"
Jul 17 22:23:58.185293 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:58.185268    2565 scope.go:117] "RemoveContainer" containerID="575bc341001116e8ce68154e800f151a9d6dc127a4265e54b9dd6f26574ee307"
Jul 17 22:23:58.185704 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:23:58.185521    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq_odh-ai-gateway-infra(063d1362-cbe3-4a75-b278-48f804a067b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq" podUID="063d1362-cbe3-4a75-b278-48f804a067b5"
Jul 17 22:23:58.264856 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:58.264825    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg"]
Jul 17 22:23:58.266952 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:23:58.266927    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4f00e5_2350_4741_ab28_31e9a16ff3a5.slice/crio-3fd5c8645822cd0a0f8b36fcbdcbc0ca35bd4143b8c826894392753f178c1048 WatchSource:0}: Error finding container 3fd5c8645822cd0a0f8b36fcbdcbc0ca35bd4143b8c826894392753f178c1048: Status 404 returned error can't find the container with id 3fd5c8645822cd0a0f8b36fcbdcbc0ca35bd4143b8c826894392753f178c1048
Jul 17 22:23:59.191006 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:59.190969    2565 generic.go:356] "Generic (PLEG): container finished" podID="ed4f00e5-2350-4741-ab28-31e9a16ff3a5" containerID="d874f5f00ed03c2dfb8f7c4671c9e358891391fd7592cc72e47dbe24702354a1" exitCode=6
Jul 17 22:23:59.191456 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:59.191059    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg" event={"ID":"ed4f00e5-2350-4741-ab28-31e9a16ff3a5","Type":"ContainerDied","Data":"d874f5f00ed03c2dfb8f7c4671c9e358891391fd7592cc72e47dbe24702354a1"}
Jul 17 22:23:59.191456 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:59.191131    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg" event={"ID":"ed4f00e5-2350-4741-ab28-31e9a16ff3a5","Type":"ContainerStarted","Data":"3fd5c8645822cd0a0f8b36fcbdcbc0ca35bd4143b8c826894392753f178c1048"}
Jul 17 22:23:59.191456 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:59.191247    2565 scope.go:117] "RemoveContainer" containerID="d874f5f00ed03c2dfb8f7c4671c9e358891391fd7592cc72e47dbe24702354a1"
Jul 17 22:23:59.193020 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:59.192998    2565 generic.go:356] "Generic (PLEG): container finished" podID="7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab" containerID="91c66f966f4f6db425f3f78af9a05a9348c91b1dceed3aedbf5277042e917767" exitCode=7
Jul 17 22:23:59.193149 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:59.193037    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv" event={"ID":"7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab","Type":"ContainerDied","Data":"91c66f966f4f6db425f3f78af9a05a9348c91b1dceed3aedbf5277042e917767"}
Jul 17 22:23:59.193149 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:59.193106    2565 scope.go:117] "RemoveContainer" containerID="6954f4f10dd4a8d5e69b835063d048ce485493f79e6c6b31188ce508a8e96eaf"
Jul 17 22:23:59.193433 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:23:59.193412    2565 scope.go:117] "RemoveContainer" containerID="91c66f966f4f6db425f3f78af9a05a9348c91b1dceed3aedbf5277042e917767"
Jul 17 22:23:59.193836 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:23:59.193620    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv_odh-ai-gateway-infra(7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv" podUID="7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab"
Jul 17 22:24:00.202455 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:00.202355    2565 generic.go:356] "Generic (PLEG): container finished" podID="ed4f00e5-2350-4741-ab28-31e9a16ff3a5" containerID="1f40bb48060ccea9451b0b6b8a794e9c7cc6c3b060b439c6dfa27054f37e623f" exitCode=6
Jul 17 22:24:00.202455 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:00.202430    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg" event={"ID":"ed4f00e5-2350-4741-ab28-31e9a16ff3a5","Type":"ContainerDied","Data":"1f40bb48060ccea9451b0b6b8a794e9c7cc6c3b060b439c6dfa27054f37e623f"}
Jul 17 22:24:00.202971 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:00.202481    2565 scope.go:117] "RemoveContainer" containerID="d874f5f00ed03c2dfb8f7c4671c9e358891391fd7592cc72e47dbe24702354a1"
Jul 17 22:24:00.202971 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:00.202654    2565 scope.go:117] "RemoveContainer" containerID="1f40bb48060ccea9451b0b6b8a794e9c7cc6c3b060b439c6dfa27054f37e623f"
Jul 17 22:24:00.202971 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:24:00.202917    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg_odh-ai-gateway-infra(ed4f00e5-2350-4741-ab28-31e9a16ff3a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg" podUID="ed4f00e5-2350-4741-ab28-31e9a16ff3a5"
Jul 17 22:24:00.235430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:00.235393    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv"]
Jul 17 22:24:00.377192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:00.377161    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv"
Jul 17 22:24:00.468950 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:00.468865    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab-maas-api-service-ca\") pod \"7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab\" (UID: \"7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab\") "
Jul 17 22:24:00.469223 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:00.469202    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab" (UID: "7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:24:00.570213 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:00.570176    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:24:01.211529 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:01.211501    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv"
Jul 17 22:24:01.211529 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:01.211534    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv" event={"ID":"7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab","Type":"ContainerDied","Data":"86555ca75d50422986b23c24a927c42f1ee6693b3f6cf0080e90cbe0c411fd89"}
Jul 17 22:24:01.212102 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:01.211598    2565 scope.go:117] "RemoveContainer" containerID="91c66f966f4f6db425f3f78af9a05a9348c91b1dceed3aedbf5277042e917767"
Jul 17 22:24:01.213420 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:01.213406    2565 scope.go:117] "RemoveContainer" containerID="1f40bb48060ccea9451b0b6b8a794e9c7cc6c3b060b439c6dfa27054f37e623f"
Jul 17 22:24:01.213630 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:24:01.213594    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg_odh-ai-gateway-infra(ed4f00e5-2350-4741-ab28-31e9a16ff3a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg" podUID="ed4f00e5-2350-4741-ab28-31e9a16ff3a5"
Jul 17 22:24:01.248945 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:01.248908    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv"]
Jul 17 22:24:01.257620 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:01.257594    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-wk4sv"]
Jul 17 22:24:03.183418 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:03.183382    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab" path="/var/lib/kubelet/pods/7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab/volumes"
Jul 17 22:24:10.176527 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:10.176493    2565 scope.go:117] "RemoveContainer" containerID="575bc341001116e8ce68154e800f151a9d6dc127a4265e54b9dd6f26574ee307"
Jul 17 22:24:11.266739 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:11.266708    2565 generic.go:356] "Generic (PLEG): container finished" podID="063d1362-cbe3-4a75-b278-48f804a067b5" containerID="94fbcb2d2bb0c8974135ddf784db150c4170678d6393b38d0a693029083c7bae" exitCode=6
Jul 17 22:24:11.267220 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:11.266786    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq" event={"ID":"063d1362-cbe3-4a75-b278-48f804a067b5","Type":"ContainerDied","Data":"94fbcb2d2bb0c8974135ddf784db150c4170678d6393b38d0a693029083c7bae"}
Jul 17 22:24:11.267220 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:11.266848    2565 scope.go:117] "RemoveContainer" containerID="575bc341001116e8ce68154e800f151a9d6dc127a4265e54b9dd6f26574ee307"
Jul 17 22:24:11.267220 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:11.267126    2565 scope.go:117] "RemoveContainer" containerID="94fbcb2d2bb0c8974135ddf784db150c4170678d6393b38d0a693029083c7bae"
Jul 17 22:24:11.267429 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:24:11.267404    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq_odh-ai-gateway-infra(063d1362-cbe3-4a75-b278-48f804a067b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq" podUID="063d1362-cbe3-4a75-b278-48f804a067b5"
Jul 17 22:24:12.303894 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:12.303859    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq"]
Jul 17 22:24:12.445143 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:12.445113    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq"
Jul 17 22:24:12.580102 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:12.579972    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/063d1362-cbe3-4a75-b278-48f804a067b5-maas-api-service-ca\") pod \"063d1362-cbe3-4a75-b278-48f804a067b5\" (UID: \"063d1362-cbe3-4a75-b278-48f804a067b5\") "
Jul 17 22:24:12.580352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:12.580330    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063d1362-cbe3-4a75-b278-48f804a067b5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "063d1362-cbe3-4a75-b278-48f804a067b5" (UID: "063d1362-cbe3-4a75-b278-48f804a067b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:24:12.681542 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:12.681500    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/063d1362-cbe3-4a75-b278-48f804a067b5-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:24:13.278638 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:13.278606    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq"
Jul 17 22:24:13.278638 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:13.278632    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq" event={"ID":"063d1362-cbe3-4a75-b278-48f804a067b5","Type":"ContainerDied","Data":"7d10e96d9a408aa9623c159ac3da7d99e832798abc8f14717c9fe29c1aa7da95"}
Jul 17 22:24:13.278875 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:13.278693    2565 scope.go:117] "RemoveContainer" containerID="94fbcb2d2bb0c8974135ddf784db150c4170678d6393b38d0a693029083c7bae"
Jul 17 22:24:13.295582 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:13.295555    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq"]
Jul 17 22:24:13.302250 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:13.302218    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8gxzq"]
Jul 17 22:24:15.176836 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:15.176804    2565 scope.go:117] "RemoveContainer" containerID="1f40bb48060ccea9451b0b6b8a794e9c7cc6c3b060b439c6dfa27054f37e623f"
Jul 17 22:24:15.183329 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:15.183306    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="063d1362-cbe3-4a75-b278-48f804a067b5" path="/var/lib/kubelet/pods/063d1362-cbe3-4a75-b278-48f804a067b5/volumes"
Jul 17 22:24:16.011566 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.011485    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr"]
Jul 17 22:24:16.011844 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.011832    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab" containerName="revoke-keys"
Jul 17 22:24:16.011898 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.011845    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab" containerName="revoke-keys"
Jul 17 22:24:16.011898 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.011856    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab" containerName="revoke-keys"
Jul 17 22:24:16.011898 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.011861    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab" containerName="revoke-keys"
Jul 17 22:24:16.011898 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.011871    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="063d1362-cbe3-4a75-b278-48f804a067b5" containerName="revoke-keys"
Jul 17 22:24:16.011898 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.011876    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="063d1362-cbe3-4a75-b278-48f804a067b5" containerName="revoke-keys"
Jul 17 22:24:16.011898 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.011894    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="063d1362-cbe3-4a75-b278-48f804a067b5" containerName="revoke-keys"
Jul 17 22:24:16.011898 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.011899    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="063d1362-cbe3-4a75-b278-48f804a067b5" containerName="revoke-keys"
Jul 17 22:24:16.012159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.011945    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="063d1362-cbe3-4a75-b278-48f804a067b5" containerName="revoke-keys"
Jul 17 22:24:16.012159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.011954    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab" containerName="revoke-keys"
Jul 17 22:24:16.012159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.011961    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab" containerName="revoke-keys"
Jul 17 22:24:16.012159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.011968    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="063d1362-cbe3-4a75-b278-48f804a067b5" containerName="revoke-keys"
Jul 17 22:24:16.014990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.014974    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr"
Jul 17 22:24:16.021824 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.021791    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr"]
Jul 17 22:24:16.112429 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.112388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d865a720-3329-4d81-a3ad-8e5839d62385-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr\" (UID: \"d865a720-3329-4d81-a3ad-8e5839d62385\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr"
Jul 17 22:24:16.213393 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.213339    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d865a720-3329-4d81-a3ad-8e5839d62385-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr\" (UID: \"d865a720-3329-4d81-a3ad-8e5839d62385\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr"
Jul 17 22:24:16.214031 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.214010    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d865a720-3329-4d81-a3ad-8e5839d62385-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr\" (UID: \"d865a720-3329-4d81-a3ad-8e5839d62385\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr"
Jul 17 22:24:16.295218 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.295118    2565 generic.go:356] "Generic (PLEG): container finished" podID="ed4f00e5-2350-4741-ab28-31e9a16ff3a5" containerID="003fc20ca62dd289db96ff5ec6d874bb754cebadcda3ad57e166ccc5c9441fe7" exitCode=6
Jul 17 22:24:16.295367 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.295208    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg" event={"ID":"ed4f00e5-2350-4741-ab28-31e9a16ff3a5","Type":"ContainerDied","Data":"003fc20ca62dd289db96ff5ec6d874bb754cebadcda3ad57e166ccc5c9441fe7"}
Jul 17 22:24:16.295367 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.295284    2565 scope.go:117] "RemoveContainer" containerID="1f40bb48060ccea9451b0b6b8a794e9c7cc6c3b060b439c6dfa27054f37e623f"
Jul 17 22:24:16.295607 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.295593    2565 scope.go:117] "RemoveContainer" containerID="003fc20ca62dd289db96ff5ec6d874bb754cebadcda3ad57e166ccc5c9441fe7"
Jul 17 22:24:16.295828 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:24:16.295810    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg_odh-ai-gateway-infra(ed4f00e5-2350-4741-ab28-31e9a16ff3a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg" podUID="ed4f00e5-2350-4741-ab28-31e9a16ff3a5"
Jul 17 22:24:16.328325 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.328291    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr"
Jul 17 22:24:16.470315 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:16.470289    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr"]
Jul 17 22:24:16.472403 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:24:16.472380    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd865a720_3329_4d81_a3ad_8e5839d62385.slice/crio-d7f224012fca36747bfef493ccae00cf2a37489f694778cb0beaff58144f9c09 WatchSource:0}: Error finding container d7f224012fca36747bfef493ccae00cf2a37489f694778cb0beaff58144f9c09: Status 404 returned error can't find the container with id d7f224012fca36747bfef493ccae00cf2a37489f694778cb0beaff58144f9c09
Jul 17 22:24:17.303173 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:17.303137    2565 generic.go:356] "Generic (PLEG): container finished" podID="d865a720-3329-4d81-a3ad-8e5839d62385" containerID="dad4bacdf2027a68d8eaca4c13d7366dc3e3f910cc00e97e0391924f0821e041" exitCode=6
Jul 17 22:24:17.303589 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:17.303222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr" event={"ID":"d865a720-3329-4d81-a3ad-8e5839d62385","Type":"ContainerDied","Data":"dad4bacdf2027a68d8eaca4c13d7366dc3e3f910cc00e97e0391924f0821e041"}
Jul 17 22:24:17.303589 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:17.303277    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr" event={"ID":"d865a720-3329-4d81-a3ad-8e5839d62385","Type":"ContainerStarted","Data":"d7f224012fca36747bfef493ccae00cf2a37489f694778cb0beaff58144f9c09"}
Jul 17 22:24:17.303589 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:17.303417    2565 scope.go:117] "RemoveContainer" containerID="dad4bacdf2027a68d8eaca4c13d7366dc3e3f910cc00e97e0391924f0821e041"
Jul 17 22:24:17.325458 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:17.325424    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg"]
Jul 17 22:24:17.479290 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:17.479260    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg"
Jul 17 22:24:17.626318 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:17.626277    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4f00e5-2350-4741-ab28-31e9a16ff3a5-maas-api-service-ca\") pod \"ed4f00e5-2350-4741-ab28-31e9a16ff3a5\" (UID: \"ed4f00e5-2350-4741-ab28-31e9a16ff3a5\") "
Jul 17 22:24:17.626640 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:17.626617    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed4f00e5-2350-4741-ab28-31e9a16ff3a5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed4f00e5-2350-4741-ab28-31e9a16ff3a5" (UID: "ed4f00e5-2350-4741-ab28-31e9a16ff3a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:24:17.727646 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:17.727605    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4f00e5-2350-4741-ab28-31e9a16ff3a5-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:24:18.309745 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:18.309711    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg"
Jul 17 22:24:18.309745 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:18.309726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg" event={"ID":"ed4f00e5-2350-4741-ab28-31e9a16ff3a5","Type":"ContainerDied","Data":"3fd5c8645822cd0a0f8b36fcbdcbc0ca35bd4143b8c826894392753f178c1048"}
Jul 17 22:24:18.310309 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:18.309791    2565 scope.go:117] "RemoveContainer" containerID="003fc20ca62dd289db96ff5ec6d874bb754cebadcda3ad57e166ccc5c9441fe7"
Jul 17 22:24:18.311863 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:18.311839    2565 generic.go:356] "Generic (PLEG): container finished" podID="d865a720-3329-4d81-a3ad-8e5839d62385" containerID="af5ddace5bf768af5add7fb1af687b6ec5c61e2cbfeb3d4094ac37b79de148a4" exitCode=6
Jul 17 22:24:18.311983 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:18.311919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr" event={"ID":"d865a720-3329-4d81-a3ad-8e5839d62385","Type":"ContainerDied","Data":"af5ddace5bf768af5add7fb1af687b6ec5c61e2cbfeb3d4094ac37b79de148a4"}
Jul 17 22:24:18.312256 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:18.312239    2565 scope.go:117] "RemoveContainer" containerID="af5ddace5bf768af5add7fb1af687b6ec5c61e2cbfeb3d4094ac37b79de148a4"
Jul 17 22:24:18.312479 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:24:18.312457    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr_odh-ai-gateway-infra(d865a720-3329-4d81-a3ad-8e5839d62385)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr" podUID="d865a720-3329-4d81-a3ad-8e5839d62385"
Jul 17 22:24:18.318711 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:18.318693    2565 scope.go:117] "RemoveContainer" containerID="dad4bacdf2027a68d8eaca4c13d7366dc3e3f910cc00e97e0391924f0821e041"
Jul 17 22:24:18.347013 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:18.346985    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg"]
Jul 17 22:24:18.358821 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:18.358789    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-d9ngg"]
Jul 17 22:24:19.182882 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:19.182846    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed4f00e5-2350-4741-ab28-31e9a16ff3a5" path="/var/lib/kubelet/pods/ed4f00e5-2350-4741-ab28-31e9a16ff3a5/volumes"
Jul 17 22:24:19.318932 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:19.318886    2565 scope.go:117] "RemoveContainer" containerID="af5ddace5bf768af5add7fb1af687b6ec5c61e2cbfeb3d4094ac37b79de148a4"
Jul 17 22:24:19.319347 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:24:19.319163    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr_odh-ai-gateway-infra(d865a720-3329-4d81-a3ad-8e5839d62385)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr" podUID="d865a720-3329-4d81-a3ad-8e5839d62385"
Jul 17 22:24:30.177475 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:30.177387    2565 scope.go:117] "RemoveContainer" containerID="af5ddace5bf768af5add7fb1af687b6ec5c61e2cbfeb3d4094ac37b79de148a4"
Jul 17 22:24:31.377130 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:31.377096    2565 generic.go:356] "Generic (PLEG): container finished" podID="d865a720-3329-4d81-a3ad-8e5839d62385" containerID="ce593e61bd16e41143f3b78d7e8dae6b47a3f1083c1c5186cfac2b414b387df8" exitCode=6
Jul 17 22:24:31.377582 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:31.377100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr" event={"ID":"d865a720-3329-4d81-a3ad-8e5839d62385","Type":"ContainerDied","Data":"ce593e61bd16e41143f3b78d7e8dae6b47a3f1083c1c5186cfac2b414b387df8"}
Jul 17 22:24:31.377582 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:31.377212    2565 scope.go:117] "RemoveContainer" containerID="af5ddace5bf768af5add7fb1af687b6ec5c61e2cbfeb3d4094ac37b79de148a4"
Jul 17 22:24:31.377582 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:31.377467    2565 scope.go:117] "RemoveContainer" containerID="ce593e61bd16e41143f3b78d7e8dae6b47a3f1083c1c5186cfac2b414b387df8"
Jul 17 22:24:31.377750 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:24:31.377713    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr_odh-ai-gateway-infra(d865a720-3329-4d81-a3ad-8e5839d62385)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr" podUID="d865a720-3329-4d81-a3ad-8e5839d62385"
Jul 17 22:24:32.392879 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.392842    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx"]
Jul 17 22:24:32.393300 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.393217    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4f00e5-2350-4741-ab28-31e9a16ff3a5" containerName="revoke-keys"
Jul 17 22:24:32.393300 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.393229    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4f00e5-2350-4741-ab28-31e9a16ff3a5" containerName="revoke-keys"
Jul 17 22:24:32.393300 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.393241    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="063d1362-cbe3-4a75-b278-48f804a067b5" containerName="revoke-keys"
Jul 17 22:24:32.393300 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.393246    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="063d1362-cbe3-4a75-b278-48f804a067b5" containerName="revoke-keys"
Jul 17 22:24:32.393300 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.393259    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab" containerName="revoke-keys"
Jul 17 22:24:32.393300 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.393264    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab" containerName="revoke-keys"
Jul 17 22:24:32.393300 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.393275    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4f00e5-2350-4741-ab28-31e9a16ff3a5" containerName="revoke-keys"
Jul 17 22:24:32.393300 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.393280    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4f00e5-2350-4741-ab28-31e9a16ff3a5" containerName="revoke-keys"
Jul 17 22:24:32.393548 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.393346    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="063d1362-cbe3-4a75-b278-48f804a067b5" containerName="revoke-keys"
Jul 17 22:24:32.393548 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.393357    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed4f00e5-2350-4741-ab28-31e9a16ff3a5" containerName="revoke-keys"
Jul 17 22:24:32.393548 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.393362    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed4f00e5-2350-4741-ab28-31e9a16ff3a5" containerName="revoke-keys"
Jul 17 22:24:32.393548 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.393371    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb1add7-fe38-42d5-a3ba-c1e4b934a9ab" containerName="revoke-keys"
Jul 17 22:24:32.397554 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.397535    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx"
Jul 17 22:24:32.405710 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.405055    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx"]
Jul 17 22:24:32.409412 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.409381    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr"]
Jul 17 22:24:32.452047 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.452010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4735010-3a8e-48d9-85e2-6fd2578c72b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx\" (UID: \"a4735010-3a8e-48d9-85e2-6fd2578c72b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx"
Jul 17 22:24:32.549718 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.549686    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr"
Jul 17 22:24:32.552867 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.552847    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d865a720-3329-4d81-a3ad-8e5839d62385-maas-api-service-ca\") pod \"d865a720-3329-4d81-a3ad-8e5839d62385\" (UID: \"d865a720-3329-4d81-a3ad-8e5839d62385\") "
Jul 17 22:24:32.553000 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.552975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4735010-3a8e-48d9-85e2-6fd2578c72b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx\" (UID: \"a4735010-3a8e-48d9-85e2-6fd2578c72b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx"
Jul 17 22:24:32.553218 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.553194    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d865a720-3329-4d81-a3ad-8e5839d62385-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d865a720-3329-4d81-a3ad-8e5839d62385" (UID: "d865a720-3329-4d81-a3ad-8e5839d62385"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:24:32.553616 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.553595    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4735010-3a8e-48d9-85e2-6fd2578c72b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx\" (UID: \"a4735010-3a8e-48d9-85e2-6fd2578c72b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx"
Jul 17 22:24:32.654544 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.654448    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d865a720-3329-4d81-a3ad-8e5839d62385-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:24:32.711463 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.711430    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx"
Jul 17 22:24:32.849511 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:32.849487    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx"]
Jul 17 22:24:32.851507 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:24:32.851470    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4735010_3a8e_48d9_85e2_6fd2578c72b1.slice/crio-3fa573d37231dfd1d170a2e2c77405c0f261e651db795245047ee01da1373ea7 WatchSource:0}: Error finding container 3fa573d37231dfd1d170a2e2c77405c0f261e651db795245047ee01da1373ea7: Status 404 returned error can't find the container with id 3fa573d37231dfd1d170a2e2c77405c0f261e651db795245047ee01da1373ea7
Jul 17 22:24:33.389405 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:33.389372    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr"
Jul 17 22:24:33.389620 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:33.389411    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr" event={"ID":"d865a720-3329-4d81-a3ad-8e5839d62385","Type":"ContainerDied","Data":"d7f224012fca36747bfef493ccae00cf2a37489f694778cb0beaff58144f9c09"}
Jul 17 22:24:33.389620 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:33.389508    2565 scope.go:117] "RemoveContainer" containerID="ce593e61bd16e41143f3b78d7e8dae6b47a3f1083c1c5186cfac2b414b387df8"
Jul 17 22:24:33.391049 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:33.391013    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx" event={"ID":"a4735010-3a8e-48d9-85e2-6fd2578c72b1","Type":"ContainerStarted","Data":"df7498d47c4ae81ecfef2064437bb56927082d94871e9f72418fcc93a60ec4a3"}
Jul 17 22:24:33.391188 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:33.391081    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx" event={"ID":"a4735010-3a8e-48d9-85e2-6fd2578c72b1","Type":"ContainerStarted","Data":"3fa573d37231dfd1d170a2e2c77405c0f261e651db795245047ee01da1373ea7"}
Jul 17 22:24:33.406245 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:33.406216    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr"]
Jul 17 22:24:33.410639 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:33.410611    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tbpzr"]
Jul 17 22:24:33.424912 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:33.424850    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx" podStartSLOduration=1.424832901 podStartE2EDuration="1.424832901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:24:33.424255511 +0000 UTC m=+2336.813272175" watchObservedRunningTime="2026-07-17 22:24:33.424832901 +0000 UTC m=+2336.813849593"
Jul 17 22:24:34.398121 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:34.398082    2565 generic.go:356] "Generic (PLEG): container finished" podID="a4735010-3a8e-48d9-85e2-6fd2578c72b1" containerID="df7498d47c4ae81ecfef2064437bb56927082d94871e9f72418fcc93a60ec4a3" exitCode=7
Jul 17 22:24:34.398312 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:34.398155    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx" event={"ID":"a4735010-3a8e-48d9-85e2-6fd2578c72b1","Type":"ContainerDied","Data":"df7498d47c4ae81ecfef2064437bb56927082d94871e9f72418fcc93a60ec4a3"}
Jul 17 22:24:34.398404 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:34.398391    2565 scope.go:117] "RemoveContainer" containerID="df7498d47c4ae81ecfef2064437bb56927082d94871e9f72418fcc93a60ec4a3"
Jul 17 22:24:35.182140 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:35.182111    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d865a720-3329-4d81-a3ad-8e5839d62385" path="/var/lib/kubelet/pods/d865a720-3329-4d81-a3ad-8e5839d62385/volumes"
Jul 17 22:24:35.405674 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:35.405636    2565 generic.go:356] "Generic (PLEG): container finished" podID="a4735010-3a8e-48d9-85e2-6fd2578c72b1" containerID="4b14daa31458d98fd52418671fcd28543c0f7917c6f7584c5890ca78fc369549" exitCode=7
Jul 17 22:24:35.405853 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:35.405716    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx" event={"ID":"a4735010-3a8e-48d9-85e2-6fd2578c72b1","Type":"ContainerDied","Data":"4b14daa31458d98fd52418671fcd28543c0f7917c6f7584c5890ca78fc369549"}
Jul 17 22:24:35.405853 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:35.405797    2565 scope.go:117] "RemoveContainer" containerID="df7498d47c4ae81ecfef2064437bb56927082d94871e9f72418fcc93a60ec4a3"
Jul 17 22:24:35.406026 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:35.406011    2565 scope.go:117] "RemoveContainer" containerID="4b14daa31458d98fd52418671fcd28543c0f7917c6f7584c5890ca78fc369549"
Jul 17 22:24:35.406277 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:24:35.406257    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx_odh-ai-gateway-infra(a4735010-3a8e-48d9-85e2-6fd2578c72b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx" podUID="a4735010-3a8e-48d9-85e2-6fd2578c72b1"
Jul 17 22:24:36.412471 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:36.412443    2565 scope.go:117] "RemoveContainer" containerID="4b14daa31458d98fd52418671fcd28543c0f7917c6f7584c5890ca78fc369549"
Jul 17 22:24:36.412878 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:24:36.412661    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx_odh-ai-gateway-infra(a4735010-3a8e-48d9-85e2-6fd2578c72b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx" podUID="a4735010-3a8e-48d9-85e2-6fd2578c72b1"
Jul 17 22:24:44.516312 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.516269    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r"]
Jul 17 22:24:44.516738 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.516667    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d865a720-3329-4d81-a3ad-8e5839d62385" containerName="revoke-keys"
Jul 17 22:24:44.516738 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.516678    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d865a720-3329-4d81-a3ad-8e5839d62385" containerName="revoke-keys"
Jul 17 22:24:44.516738 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.516688    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d865a720-3329-4d81-a3ad-8e5839d62385" containerName="revoke-keys"
Jul 17 22:24:44.516738 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.516695    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d865a720-3329-4d81-a3ad-8e5839d62385" containerName="revoke-keys"
Jul 17 22:24:44.516738 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.516709    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4f00e5-2350-4741-ab28-31e9a16ff3a5" containerName="revoke-keys"
Jul 17 22:24:44.516738 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.516714    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4f00e5-2350-4741-ab28-31e9a16ff3a5" containerName="revoke-keys"
Jul 17 22:24:44.516738 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.516720    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d865a720-3329-4d81-a3ad-8e5839d62385" containerName="revoke-keys"
Jul 17 22:24:44.516738 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.516725    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d865a720-3329-4d81-a3ad-8e5839d62385" containerName="revoke-keys"
Jul 17 22:24:44.517023 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.516790    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d865a720-3329-4d81-a3ad-8e5839d62385" containerName="revoke-keys"
Jul 17 22:24:44.517023 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.516800    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed4f00e5-2350-4741-ab28-31e9a16ff3a5" containerName="revoke-keys"
Jul 17 22:24:44.517023 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.516806    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d865a720-3329-4d81-a3ad-8e5839d62385" containerName="revoke-keys"
Jul 17 22:24:44.519837 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.519820    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r"
Jul 17 22:24:44.526914 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.526878    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r"]
Jul 17 22:24:44.568693 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.568656    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7da1853-01a1-4efd-a120-e92c9a5a4572-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r\" (UID: \"b7da1853-01a1-4efd-a120-e92c9a5a4572\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r"
Jul 17 22:24:44.669851 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.669806    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7da1853-01a1-4efd-a120-e92c9a5a4572-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r\" (UID: \"b7da1853-01a1-4efd-a120-e92c9a5a4572\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r"
Jul 17 22:24:44.670511 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.670491    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7da1853-01a1-4efd-a120-e92c9a5a4572-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r\" (UID: \"b7da1853-01a1-4efd-a120-e92c9a5a4572\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r"
Jul 17 22:24:44.832603 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.832499    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r"
Jul 17 22:24:44.969875 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:44.969837    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r"]
Jul 17 22:24:44.972357 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:24:44.972321    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7da1853_01a1_4efd_a120_e92c9a5a4572.slice/crio-1b552776367a9ef2d774c41fef023687cc876e642ae681331e9d241b06dd5290 WatchSource:0}: Error finding container 1b552776367a9ef2d774c41fef023687cc876e642ae681331e9d241b06dd5290: Status 404 returned error can't find the container with id 1b552776367a9ef2d774c41fef023687cc876e642ae681331e9d241b06dd5290
Jul 17 22:24:45.465542 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:45.465446    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r" event={"ID":"b7da1853-01a1-4efd-a120-e92c9a5a4572","Type":"ContainerStarted","Data":"f8eafd6ea81d8ad34bbc7ad89ff5ce33d80feebb29e38d78176e839888ac9447"}
Jul 17 22:24:45.465542 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:45.465498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r" event={"ID":"b7da1853-01a1-4efd-a120-e92c9a5a4572","Type":"ContainerStarted","Data":"1b552776367a9ef2d774c41fef023687cc876e642ae681331e9d241b06dd5290"}
Jul 17 22:24:45.482739 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:45.482673    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r" podStartSLOduration=1.482651712 podStartE2EDuration="1.482651712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:24:45.482220563 +0000 UTC m=+2348.871237233" watchObservedRunningTime="2026-07-17 22:24:45.482651712 +0000 UTC m=+2348.871668355"
Jul 17 22:24:46.471647 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:46.471614    2565 generic.go:356] "Generic (PLEG): container finished" podID="b7da1853-01a1-4efd-a120-e92c9a5a4572" containerID="f8eafd6ea81d8ad34bbc7ad89ff5ce33d80feebb29e38d78176e839888ac9447" exitCode=6
Jul 17 22:24:46.472086 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:46.471663    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r" event={"ID":"b7da1853-01a1-4efd-a120-e92c9a5a4572","Type":"ContainerDied","Data":"f8eafd6ea81d8ad34bbc7ad89ff5ce33d80feebb29e38d78176e839888ac9447"}
Jul 17 22:24:46.472086 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:46.471892    2565 scope.go:117] "RemoveContainer" containerID="f8eafd6ea81d8ad34bbc7ad89ff5ce33d80feebb29e38d78176e839888ac9447"
Jul 17 22:24:47.478121 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:47.478002    2565 generic.go:356] "Generic (PLEG): container finished" podID="b7da1853-01a1-4efd-a120-e92c9a5a4572" containerID="3946ff5c62b9dd3eb5dfeb33ebce907276fc9d7c70fb9f6a0fc07829bee3fe13" exitCode=6
Jul 17 22:24:47.478121 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:47.478056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r" event={"ID":"b7da1853-01a1-4efd-a120-e92c9a5a4572","Type":"ContainerDied","Data":"3946ff5c62b9dd3eb5dfeb33ebce907276fc9d7c70fb9f6a0fc07829bee3fe13"}
Jul 17 22:24:47.478642 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:47.478127    2565 scope.go:117] "RemoveContainer" containerID="f8eafd6ea81d8ad34bbc7ad89ff5ce33d80feebb29e38d78176e839888ac9447"
Jul 17 22:24:47.478642 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:47.478338    2565 scope.go:117] "RemoveContainer" containerID="3946ff5c62b9dd3eb5dfeb33ebce907276fc9d7c70fb9f6a0fc07829bee3fe13"
Jul 17 22:24:47.478642 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:24:47.478603    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r_odh-ai-gateway-infra(b7da1853-01a1-4efd-a120-e92c9a5a4572)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r" podUID="b7da1853-01a1-4efd-a120-e92c9a5a4572"
Jul 17 22:24:48.484803 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:48.484771    2565 scope.go:117] "RemoveContainer" containerID="3946ff5c62b9dd3eb5dfeb33ebce907276fc9d7c70fb9f6a0fc07829bee3fe13"
Jul 17 22:24:48.485229 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:24:48.484980    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r_odh-ai-gateway-infra(b7da1853-01a1-4efd-a120-e92c9a5a4572)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r" podUID="b7da1853-01a1-4efd-a120-e92c9a5a4572"
Jul 17 22:24:52.176378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:52.176341    2565 scope.go:117] "RemoveContainer" containerID="4b14daa31458d98fd52418671fcd28543c0f7917c6f7584c5890ca78fc369549"
Jul 17 22:24:52.504664 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:52.504628    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx" event={"ID":"a4735010-3a8e-48d9-85e2-6fd2578c72b1","Type":"ContainerStarted","Data":"ea2d480d8532a138504dc921576ce0a3c404e9a9a74e50080f32ba11ada490d5"}
Jul 17 22:24:53.511046 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:53.511011    2565 generic.go:356] "Generic (PLEG): container finished" podID="a4735010-3a8e-48d9-85e2-6fd2578c72b1" containerID="ea2d480d8532a138504dc921576ce0a3c404e9a9a74e50080f32ba11ada490d5" exitCode=7
Jul 17 22:24:53.511485 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:53.511105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx" event={"ID":"a4735010-3a8e-48d9-85e2-6fd2578c72b1","Type":"ContainerDied","Data":"ea2d480d8532a138504dc921576ce0a3c404e9a9a74e50080f32ba11ada490d5"}
Jul 17 22:24:53.511485 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:53.511170    2565 scope.go:117] "RemoveContainer" containerID="4b14daa31458d98fd52418671fcd28543c0f7917c6f7584c5890ca78fc369549"
Jul 17 22:24:53.511485 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:53.511455    2565 scope.go:117] "RemoveContainer" containerID="ea2d480d8532a138504dc921576ce0a3c404e9a9a74e50080f32ba11ada490d5"
Jul 17 22:24:53.511686 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:24:53.511663    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx_odh-ai-gateway-infra(a4735010-3a8e-48d9-85e2-6fd2578c72b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx" podUID="a4735010-3a8e-48d9-85e2-6fd2578c72b1"
Jul 17 22:24:53.536296 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:53.536263    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx"]
Jul 17 22:24:54.661113 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:54.661088    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx"
Jul 17 22:24:54.771188 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:54.771100    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4735010-3a8e-48d9-85e2-6fd2578c72b1-maas-api-service-ca\") pod \"a4735010-3a8e-48d9-85e2-6fd2578c72b1\" (UID: \"a4735010-3a8e-48d9-85e2-6fd2578c72b1\") "
Jul 17 22:24:54.771452 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:54.771427    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4735010-3a8e-48d9-85e2-6fd2578c72b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a4735010-3a8e-48d9-85e2-6fd2578c72b1" (UID: "a4735010-3a8e-48d9-85e2-6fd2578c72b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:24:54.872531 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:54.872492    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4735010-3a8e-48d9-85e2-6fd2578c72b1-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:24:55.524094 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:55.524035    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx" event={"ID":"a4735010-3a8e-48d9-85e2-6fd2578c72b1","Type":"ContainerDied","Data":"3fa573d37231dfd1d170a2e2c77405c0f261e651db795245047ee01da1373ea7"}
Jul 17 22:24:55.524094 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:55.524059    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx"
Jul 17 22:24:55.524346 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:55.524156    2565 scope.go:117] "RemoveContainer" containerID="ea2d480d8532a138504dc921576ce0a3c404e9a9a74e50080f32ba11ada490d5"
Jul 17 22:24:55.539906 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:55.539878    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx"]
Jul 17 22:24:55.544355 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:55.544328    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-cmxmx"]
Jul 17 22:24:57.181968 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:57.181932    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4735010-3a8e-48d9-85e2-6fd2578c72b1" path="/var/lib/kubelet/pods/a4735010-3a8e-48d9-85e2-6fd2578c72b1/volumes"
Jul 17 22:24:57.848257 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:57.848222    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5"]
Jul 17 22:24:57.848618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:57.848606    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4735010-3a8e-48d9-85e2-6fd2578c72b1" containerName="revoke-keys"
Jul 17 22:24:57.848659 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:57.848621    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4735010-3a8e-48d9-85e2-6fd2578c72b1" containerName="revoke-keys"
Jul 17 22:24:57.848659 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:57.848644    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4735010-3a8e-48d9-85e2-6fd2578c72b1" containerName="revoke-keys"
Jul 17 22:24:57.848659 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:57.848650    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4735010-3a8e-48d9-85e2-6fd2578c72b1" containerName="revoke-keys"
Jul 17 22:24:57.848781 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:57.848708    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4735010-3a8e-48d9-85e2-6fd2578c72b1" containerName="revoke-keys"
Jul 17 22:24:57.848781 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:57.848720    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d865a720-3329-4d81-a3ad-8e5839d62385" containerName="revoke-keys"
Jul 17 22:24:57.848781 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:57.848726    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4735010-3a8e-48d9-85e2-6fd2578c72b1" containerName="revoke-keys"
Jul 17 22:24:57.852990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:57.852970    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5"
Jul 17 22:24:57.858745 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:57.858706    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5"]
Jul 17 22:24:58.005389 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:58.005353    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a66e6bad-988d-4b38-9b17-f830d5cb462d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5\" (UID: \"a66e6bad-988d-4b38-9b17-f830d5cb462d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5"
Jul 17 22:24:58.106861 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:58.106767    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a66e6bad-988d-4b38-9b17-f830d5cb462d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5\" (UID: \"a66e6bad-988d-4b38-9b17-f830d5cb462d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5"
Jul 17 22:24:58.107486 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:58.107467    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a66e6bad-988d-4b38-9b17-f830d5cb462d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5\" (UID: \"a66e6bad-988d-4b38-9b17-f830d5cb462d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5"
Jul 17 22:24:58.165838 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:58.165799    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5"
Jul 17 22:24:58.301660 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:58.301631    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5"]
Jul 17 22:24:58.303222 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:24:58.303180    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66e6bad_988d_4b38_9b17_f830d5cb462d.slice/crio-7ad6e009846545a591e7190b6cbf26e6a7dbdbed198ad92e5a866ebb2bc43885 WatchSource:0}: Error finding container 7ad6e009846545a591e7190b6cbf26e6a7dbdbed198ad92e5a866ebb2bc43885: Status 404 returned error can't find the container with id 7ad6e009846545a591e7190b6cbf26e6a7dbdbed198ad92e5a866ebb2bc43885
Jul 17 22:24:58.539889 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:58.539853    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5" event={"ID":"a66e6bad-988d-4b38-9b17-f830d5cb462d","Type":"ContainerStarted","Data":"7ad6e009846545a591e7190b6cbf26e6a7dbdbed198ad92e5a866ebb2bc43885"}
Jul 17 22:24:59.546681 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:59.546647    2565 generic.go:356] "Generic (PLEG): container finished" podID="a66e6bad-988d-4b38-9b17-f830d5cb462d" containerID="3da6a23af4a6b1a01e119e6beac15a5267b329ab441a816f44ef9e2a7eaefc40" exitCode=6
Jul 17 22:24:59.547207 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:59.546700    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5" event={"ID":"a66e6bad-988d-4b38-9b17-f830d5cb462d","Type":"ContainerDied","Data":"3da6a23af4a6b1a01e119e6beac15a5267b329ab441a816f44ef9e2a7eaefc40"}
Jul 17 22:24:59.547207 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:24:59.546933    2565 scope.go:117] "RemoveContainer" containerID="3da6a23af4a6b1a01e119e6beac15a5267b329ab441a816f44ef9e2a7eaefc40"
Jul 17 22:25:00.177261 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:00.177187    2565 scope.go:117] "RemoveContainer" containerID="3946ff5c62b9dd3eb5dfeb33ebce907276fc9d7c70fb9f6a0fc07829bee3fe13"
Jul 17 22:25:00.553086 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:00.553039    2565 generic.go:356] "Generic (PLEG): container finished" podID="a66e6bad-988d-4b38-9b17-f830d5cb462d" containerID="f04d8158c04a9f3ca2788401099483ef14849617181f23f1e911fe6b49005da4" exitCode=6
Jul 17 22:25:00.553534 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:00.553126    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5" event={"ID":"a66e6bad-988d-4b38-9b17-f830d5cb462d","Type":"ContainerDied","Data":"f04d8158c04a9f3ca2788401099483ef14849617181f23f1e911fe6b49005da4"}
Jul 17 22:25:00.553534 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:00.553233    2565 scope.go:117] "RemoveContainer" containerID="3da6a23af4a6b1a01e119e6beac15a5267b329ab441a816f44ef9e2a7eaefc40"
Jul 17 22:25:00.553534 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:00.553407    2565 scope.go:117] "RemoveContainer" containerID="f04d8158c04a9f3ca2788401099483ef14849617181f23f1e911fe6b49005da4"
Jul 17 22:25:00.553681 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:25:00.553649    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5_odh-ai-gateway-infra(a66e6bad-988d-4b38-9b17-f830d5cb462d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5" podUID="a66e6bad-988d-4b38-9b17-f830d5cb462d"
Jul 17 22:25:00.555111 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:00.555059    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r" event={"ID":"b7da1853-01a1-4efd-a120-e92c9a5a4572","Type":"ContainerStarted","Data":"c95c7062641c3e9d667bef63ef084ec23132cded661c1bd97f9bef617931e644"}
Jul 17 22:25:01.562376 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:01.562342    2565 scope.go:117] "RemoveContainer" containerID="f04d8158c04a9f3ca2788401099483ef14849617181f23f1e911fe6b49005da4"
Jul 17 22:25:01.562915 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:25:01.562596    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5_odh-ai-gateway-infra(a66e6bad-988d-4b38-9b17-f830d5cb462d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5" podUID="a66e6bad-988d-4b38-9b17-f830d5cb462d"
Jul 17 22:25:01.563813 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:01.563785    2565 generic.go:356] "Generic (PLEG): container finished" podID="b7da1853-01a1-4efd-a120-e92c9a5a4572" containerID="c95c7062641c3e9d667bef63ef084ec23132cded661c1bd97f9bef617931e644" exitCode=6
Jul 17 22:25:01.563954 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:01.563830    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r" event={"ID":"b7da1853-01a1-4efd-a120-e92c9a5a4572","Type":"ContainerDied","Data":"c95c7062641c3e9d667bef63ef084ec23132cded661c1bd97f9bef617931e644"}
Jul 17 22:25:01.563954 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:01.563877    2565 scope.go:117] "RemoveContainer" containerID="3946ff5c62b9dd3eb5dfeb33ebce907276fc9d7c70fb9f6a0fc07829bee3fe13"
Jul 17 22:25:01.564168 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:01.564154    2565 scope.go:117] "RemoveContainer" containerID="c95c7062641c3e9d667bef63ef084ec23132cded661c1bd97f9bef617931e644"
Jul 17 22:25:01.564350 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:25:01.564331    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r_odh-ai-gateway-infra(b7da1853-01a1-4efd-a120-e92c9a5a4572)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r" podUID="b7da1853-01a1-4efd-a120-e92c9a5a4572"
Jul 17 22:25:01.593161 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:01.593124    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r"]
Jul 17 22:25:02.712190 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:02.712162    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r"
Jul 17 22:25:02.849516 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:02.849417    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7da1853-01a1-4efd-a120-e92c9a5a4572-maas-api-service-ca\") pod \"b7da1853-01a1-4efd-a120-e92c9a5a4572\" (UID: \"b7da1853-01a1-4efd-a120-e92c9a5a4572\") "
Jul 17 22:25:02.849770 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:02.849747    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7da1853-01a1-4efd-a120-e92c9a5a4572-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b7da1853-01a1-4efd-a120-e92c9a5a4572" (UID: "b7da1853-01a1-4efd-a120-e92c9a5a4572"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:25:02.950078 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:02.950028    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7da1853-01a1-4efd-a120-e92c9a5a4572-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:25:03.576206 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:03.576172    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r"
Jul 17 22:25:03.576206 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:03.576201    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r" event={"ID":"b7da1853-01a1-4efd-a120-e92c9a5a4572","Type":"ContainerDied","Data":"1b552776367a9ef2d774c41fef023687cc876e642ae681331e9d241b06dd5290"}
Jul 17 22:25:03.576441 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:03.576273    2565 scope.go:117] "RemoveContainer" containerID="c95c7062641c3e9d667bef63ef084ec23132cded661c1bd97f9bef617931e644"
Jul 17 22:25:03.594232 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:03.594198    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r"]
Jul 17 22:25:03.606919 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:03.606887    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-bzf2r"]
Jul 17 22:25:05.184060 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:05.184024    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7da1853-01a1-4efd-a120-e92c9a5a4572" path="/var/lib/kubelet/pods/b7da1853-01a1-4efd-a120-e92c9a5a4572/volumes"
Jul 17 22:25:06.053918 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.053878    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr"]
Jul 17 22:25:06.054322 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.054306    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7da1853-01a1-4efd-a120-e92c9a5a4572" containerName="revoke-keys"
Jul 17 22:25:06.054369 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.054323    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7da1853-01a1-4efd-a120-e92c9a5a4572" containerName="revoke-keys"
Jul 17 22:25:06.054369 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.054341    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4735010-3a8e-48d9-85e2-6fd2578c72b1" containerName="revoke-keys"
Jul 17 22:25:06.054369 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.054346    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4735010-3a8e-48d9-85e2-6fd2578c72b1" containerName="revoke-keys"
Jul 17 22:25:06.054369 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.054359    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7da1853-01a1-4efd-a120-e92c9a5a4572" containerName="revoke-keys"
Jul 17 22:25:06.054369 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.054364    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7da1853-01a1-4efd-a120-e92c9a5a4572" containerName="revoke-keys"
Jul 17 22:25:06.054369 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.054370    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7da1853-01a1-4efd-a120-e92c9a5a4572" containerName="revoke-keys"
Jul 17 22:25:06.054562 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.054375    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7da1853-01a1-4efd-a120-e92c9a5a4572" containerName="revoke-keys"
Jul 17 22:25:06.054562 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.054432    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7da1853-01a1-4efd-a120-e92c9a5a4572" containerName="revoke-keys"
Jul 17 22:25:06.054562 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.054441    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7da1853-01a1-4efd-a120-e92c9a5a4572" containerName="revoke-keys"
Jul 17 22:25:06.054562 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.054447    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4735010-3a8e-48d9-85e2-6fd2578c72b1" containerName="revoke-keys"
Jul 17 22:25:06.058866 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.058849    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr"
Jul 17 22:25:06.064635 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.064609    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr"]
Jul 17 22:25:06.186293 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.186249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2fdd9c-51fc-4f24-bf1b-b968d8674010-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr\" (UID: \"ec2fdd9c-51fc-4f24-bf1b-b968d8674010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr"
Jul 17 22:25:06.286814 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.286773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2fdd9c-51fc-4f24-bf1b-b968d8674010-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr\" (UID: \"ec2fdd9c-51fc-4f24-bf1b-b968d8674010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr"
Jul 17 22:25:06.287423 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.287403    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2fdd9c-51fc-4f24-bf1b-b968d8674010-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr\" (UID: \"ec2fdd9c-51fc-4f24-bf1b-b968d8674010\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr"
Jul 17 22:25:06.371164 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.371056    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr"
Jul 17 22:25:06.504763 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.504590    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr"]
Jul 17 22:25:06.507139 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:25:06.507109    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2fdd9c_51fc_4f24_bf1b_b968d8674010.slice/crio-07803cb24b750a30fab451d83e203ee2310cda653e11df5c55d7c40b6cac402d WatchSource:0}: Error finding container 07803cb24b750a30fab451d83e203ee2310cda653e11df5c55d7c40b6cac402d: Status 404 returned error can't find the container with id 07803cb24b750a30fab451d83e203ee2310cda653e11df5c55d7c40b6cac402d
Jul 17 22:25:06.593659 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:06.593621    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr" event={"ID":"ec2fdd9c-51fc-4f24-bf1b-b968d8674010","Type":"ContainerStarted","Data":"07803cb24b750a30fab451d83e203ee2310cda653e11df5c55d7c40b6cac402d"}
Jul 17 22:25:07.600384 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:07.600348    2565 generic.go:356] "Generic (PLEG): container finished" podID="ec2fdd9c-51fc-4f24-bf1b-b968d8674010" containerID="eb69ef22373cb20d9fc2c42f531f336dc976591211cdcf9e3eb6a9e76ce68d18" exitCode=6
Jul 17 22:25:07.600813 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:07.600404    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr" event={"ID":"ec2fdd9c-51fc-4f24-bf1b-b968d8674010","Type":"ContainerDied","Data":"eb69ef22373cb20d9fc2c42f531f336dc976591211cdcf9e3eb6a9e76ce68d18"}
Jul 17 22:25:07.600813 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:07.600617    2565 scope.go:117] "RemoveContainer" containerID="eb69ef22373cb20d9fc2c42f531f336dc976591211cdcf9e3eb6a9e76ce68d18"
Jul 17 22:25:08.606910 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:08.606875    2565 generic.go:356] "Generic (PLEG): container finished" podID="ec2fdd9c-51fc-4f24-bf1b-b968d8674010" containerID="231cb79e69f1d21a5fef79c66f7be58808bfe292750f6ca28bd79e378ae06c49" exitCode=6
Jul 17 22:25:08.607370 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:08.606939    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr" event={"ID":"ec2fdd9c-51fc-4f24-bf1b-b968d8674010","Type":"ContainerDied","Data":"231cb79e69f1d21a5fef79c66f7be58808bfe292750f6ca28bd79e378ae06c49"}
Jul 17 22:25:08.607370 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:08.606985    2565 scope.go:117] "RemoveContainer" containerID="eb69ef22373cb20d9fc2c42f531f336dc976591211cdcf9e3eb6a9e76ce68d18"
Jul 17 22:25:08.607370 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:08.607151    2565 scope.go:117] "RemoveContainer" containerID="231cb79e69f1d21a5fef79c66f7be58808bfe292750f6ca28bd79e378ae06c49"
Jul 17 22:25:08.607506 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:25:08.607389    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr_odh-ai-gateway-infra(ec2fdd9c-51fc-4f24-bf1b-b968d8674010)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr" podUID="ec2fdd9c-51fc-4f24-bf1b-b968d8674010"
Jul 17 22:25:09.613767 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:09.613732    2565 scope.go:117] "RemoveContainer" containerID="231cb79e69f1d21a5fef79c66f7be58808bfe292750f6ca28bd79e378ae06c49"
Jul 17 22:25:09.614164 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:25:09.613980    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr_odh-ai-gateway-infra(ec2fdd9c-51fc-4f24-bf1b-b968d8674010)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr" podUID="ec2fdd9c-51fc-4f24-bf1b-b968d8674010"
Jul 17 22:25:16.181858 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:16.181820    2565 scope.go:117] "RemoveContainer" containerID="f04d8158c04a9f3ca2788401099483ef14849617181f23f1e911fe6b49005da4"
Jul 17 22:25:16.648767 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:16.648732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5" event={"ID":"a66e6bad-988d-4b38-9b17-f830d5cb462d","Type":"ContainerStarted","Data":"12e016db58cc2f7e75885bfbedf427088390edd95195ad7685fb8cd7b6b7ef9b"}
Jul 17 22:25:16.666055 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:16.666003    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5" podStartSLOduration=19.665979934 podStartE2EDuration="19.665979934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:25:16.66543071 +0000 UTC m=+2380.054447363" watchObservedRunningTime="2026-07-17 22:25:16.665979934 +0000 UTC m=+2380.054996606"
Jul 17 22:25:17.655210 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:17.655177    2565 generic.go:356] "Generic (PLEG): container finished" podID="a66e6bad-988d-4b38-9b17-f830d5cb462d" containerID="12e016db58cc2f7e75885bfbedf427088390edd95195ad7685fb8cd7b6b7ef9b" exitCode=6
Jul 17 22:25:17.655645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:17.655228    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5" event={"ID":"a66e6bad-988d-4b38-9b17-f830d5cb462d","Type":"ContainerDied","Data":"12e016db58cc2f7e75885bfbedf427088390edd95195ad7685fb8cd7b6b7ef9b"}
Jul 17 22:25:17.655645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:17.655276    2565 scope.go:117] "RemoveContainer" containerID="f04d8158c04a9f3ca2788401099483ef14849617181f23f1e911fe6b49005da4"
Jul 17 22:25:17.655645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:17.655615    2565 scope.go:117] "RemoveContainer" containerID="12e016db58cc2f7e75885bfbedf427088390edd95195ad7685fb8cd7b6b7ef9b"
Jul 17 22:25:17.655878 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:25:17.655859    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5_odh-ai-gateway-infra(a66e6bad-988d-4b38-9b17-f830d5cb462d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5" podUID="a66e6bad-988d-4b38-9b17-f830d5cb462d"
Jul 17 22:25:17.675905 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:17.675874    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5"]
Jul 17 22:25:18.525355 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:18.525320    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:25:18.800558 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:18.800527    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5"
Jul 17 22:25:18.905214 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:18.905179    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a66e6bad-988d-4b38-9b17-f830d5cb462d-maas-api-service-ca\") pod \"a66e6bad-988d-4b38-9b17-f830d5cb462d\" (UID: \"a66e6bad-988d-4b38-9b17-f830d5cb462d\") "
Jul 17 22:25:18.905553 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:18.905531    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a66e6bad-988d-4b38-9b17-f830d5cb462d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a66e6bad-988d-4b38-9b17-f830d5cb462d" (UID: "a66e6bad-988d-4b38-9b17-f830d5cb462d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:25:19.006491 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:19.006461    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a66e6bad-988d-4b38-9b17-f830d5cb462d-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:25:19.667307 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:19.667265    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5" event={"ID":"a66e6bad-988d-4b38-9b17-f830d5cb462d","Type":"ContainerDied","Data":"7ad6e009846545a591e7190b6cbf26e6a7dbdbed198ad92e5a866ebb2bc43885"}
Jul 17 22:25:19.667497 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:19.667333    2565 scope.go:117] "RemoveContainer" containerID="12e016db58cc2f7e75885bfbedf427088390edd95195ad7685fb8cd7b6b7ef9b"
Jul 17 22:25:19.667497 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:19.667290    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5"
Jul 17 22:25:19.682709 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:19.682681    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5"]
Jul 17 22:25:19.688735 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:19.688701    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zqds5"]
Jul 17 22:25:21.181989 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:21.181940    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a66e6bad-988d-4b38-9b17-f830d5cb462d" path="/var/lib/kubelet/pods/a66e6bad-988d-4b38-9b17-f830d5cb462d/volumes"
Jul 17 22:25:25.176393 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:25.176359    2565 scope.go:117] "RemoveContainer" containerID="231cb79e69f1d21a5fef79c66f7be58808bfe292750f6ca28bd79e378ae06c49"
Jul 17 22:25:25.697488 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:25.697449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr" event={"ID":"ec2fdd9c-51fc-4f24-bf1b-b968d8674010","Type":"ContainerStarted","Data":"083d685fede2e06dda4413d118d8069a80e341269665775241bfffb783398871"}
Jul 17 22:25:25.713408 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:25.713356    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr" podStartSLOduration=19.713339904 podStartE2EDuration="19.713339904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:25:25.71212582 +0000 UTC m=+2389.101142472" watchObservedRunningTime="2026-07-17 22:25:25.713339904 +0000 UTC m=+2389.102356554"
Jul 17 22:25:26.703813 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:26.703781    2565 generic.go:356] "Generic (PLEG): container finished" podID="ec2fdd9c-51fc-4f24-bf1b-b968d8674010" containerID="083d685fede2e06dda4413d118d8069a80e341269665775241bfffb783398871" exitCode=6
Jul 17 22:25:26.704299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:26.703856    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr" event={"ID":"ec2fdd9c-51fc-4f24-bf1b-b968d8674010","Type":"ContainerDied","Data":"083d685fede2e06dda4413d118d8069a80e341269665775241bfffb783398871"}
Jul 17 22:25:26.704299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:26.703921    2565 scope.go:117] "RemoveContainer" containerID="231cb79e69f1d21a5fef79c66f7be58808bfe292750f6ca28bd79e378ae06c49"
Jul 17 22:25:26.704299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:26.704218    2565 scope.go:117] "RemoveContainer" containerID="083d685fede2e06dda4413d118d8069a80e341269665775241bfffb783398871"
Jul 17 22:25:26.704527 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:25:26.704505    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr_odh-ai-gateway-infra(ec2fdd9c-51fc-4f24-bf1b-b968d8674010)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr" podUID="ec2fdd9c-51fc-4f24-bf1b-b968d8674010"
Jul 17 22:25:26.724093 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:26.724050    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr"]
Jul 17 22:25:27.869497 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:27.869471    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr"
Jul 17 22:25:27.988210 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:27.988108    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2fdd9c-51fc-4f24-bf1b-b968d8674010-maas-api-service-ca\") pod \"ec2fdd9c-51fc-4f24-bf1b-b968d8674010\" (UID: \"ec2fdd9c-51fc-4f24-bf1b-b968d8674010\") "
Jul 17 22:25:27.988524 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:27.988501    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2fdd9c-51fc-4f24-bf1b-b968d8674010-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ec2fdd9c-51fc-4f24-bf1b-b968d8674010" (UID: "ec2fdd9c-51fc-4f24-bf1b-b968d8674010"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:25:28.089698 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:28.089663    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2fdd9c-51fc-4f24-bf1b-b968d8674010-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:25:28.716264 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:28.716236    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr"
Jul 17 22:25:28.716446 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:28.716265    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr" event={"ID":"ec2fdd9c-51fc-4f24-bf1b-b968d8674010","Type":"ContainerDied","Data":"07803cb24b750a30fab451d83e203ee2310cda653e11df5c55d7c40b6cac402d"}
Jul 17 22:25:28.716446 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:28.716334    2565 scope.go:117] "RemoveContainer" containerID="083d685fede2e06dda4413d118d8069a80e341269665775241bfffb783398871"
Jul 17 22:25:28.739681 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:28.739647    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr"]
Jul 17 22:25:28.743567 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:28.743541    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-qrgpr"]
Jul 17 22:25:29.182552 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:29.182516    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec2fdd9c-51fc-4f24-bf1b-b968d8674010" path="/var/lib/kubelet/pods/ec2fdd9c-51fc-4f24-bf1b-b968d8674010/volumes"
Jul 17 22:25:30.282327 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:30.282296    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:25:30.381349 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:30.381318    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:25:30.442300 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:30.442270    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:25:32.465319 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465284    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm"]
Jul 17 22:25:32.465731 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465660    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a66e6bad-988d-4b38-9b17-f830d5cb462d" containerName="revoke-keys"
Jul 17 22:25:32.465731 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465672    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a66e6bad-988d-4b38-9b17-f830d5cb462d" containerName="revoke-keys"
Jul 17 22:25:32.465731 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465681    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec2fdd9c-51fc-4f24-bf1b-b968d8674010" containerName="revoke-keys"
Jul 17 22:25:32.465731 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465687    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec2fdd9c-51fc-4f24-bf1b-b968d8674010" containerName="revoke-keys"
Jul 17 22:25:32.465731 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465696    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a66e6bad-988d-4b38-9b17-f830d5cb462d" containerName="revoke-keys"
Jul 17 22:25:32.465731 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465702    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a66e6bad-988d-4b38-9b17-f830d5cb462d" containerName="revoke-keys"
Jul 17 22:25:32.465731 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465710    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec2fdd9c-51fc-4f24-bf1b-b968d8674010" containerName="revoke-keys"
Jul 17 22:25:32.465731 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465715    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec2fdd9c-51fc-4f24-bf1b-b968d8674010" containerName="revoke-keys"
Jul 17 22:25:32.465731 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465723    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec2fdd9c-51fc-4f24-bf1b-b968d8674010" containerName="revoke-keys"
Jul 17 22:25:32.465731 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465727    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec2fdd9c-51fc-4f24-bf1b-b968d8674010" containerName="revoke-keys"
Jul 17 22:25:32.466045 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465785    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec2fdd9c-51fc-4f24-bf1b-b968d8674010" containerName="revoke-keys"
Jul 17 22:25:32.466045 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465793    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a66e6bad-988d-4b38-9b17-f830d5cb462d" containerName="revoke-keys"
Jul 17 22:25:32.466045 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465800    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec2fdd9c-51fc-4f24-bf1b-b968d8674010" containerName="revoke-keys"
Jul 17 22:25:32.466045 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465806    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec2fdd9c-51fc-4f24-bf1b-b968d8674010" containerName="revoke-keys"
Jul 17 22:25:32.466045 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465811    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a66e6bad-988d-4b38-9b17-f830d5cb462d" containerName="revoke-keys"
Jul 17 22:25:32.466045 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.465817    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7da1853-01a1-4efd-a120-e92c9a5a4572" containerName="revoke-keys"
Jul 17 22:25:32.470653 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.470624    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm"
Jul 17 22:25:32.473119 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.473089    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:25:32.473801 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.473778    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:25:32.475966 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.475936    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm"]
Jul 17 22:25:32.533853 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.533809    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b19412-877c-4abe-8a05-4cafed8b6b8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm\" (UID: \"83b19412-877c-4abe-8a05-4cafed8b6b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm"
Jul 17 22:25:32.634908 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.634864    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b19412-877c-4abe-8a05-4cafed8b6b8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm\" (UID: \"83b19412-877c-4abe-8a05-4cafed8b6b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm"
Jul 17 22:25:32.635551 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.635532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b19412-877c-4abe-8a05-4cafed8b6b8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm\" (UID: \"83b19412-877c-4abe-8a05-4cafed8b6b8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm"
Jul 17 22:25:32.783655 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.783614    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm"
Jul 17 22:25:32.923255 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:32.923110    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm"]
Jul 17 22:25:32.925659 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:25:32.925631    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b19412_877c_4abe_8a05_4cafed8b6b8b.slice/crio-5ed608ad316de7cbd4f1b9335ddde41c256c8ec743fb24e0ef5339d9a0353e9c WatchSource:0}: Error finding container 5ed608ad316de7cbd4f1b9335ddde41c256c8ec743fb24e0ef5339d9a0353e9c: Status 404 returned error can't find the container with id 5ed608ad316de7cbd4f1b9335ddde41c256c8ec743fb24e0ef5339d9a0353e9c
Jul 17 22:25:33.090813 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:33.090709    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:25:33.191714 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:33.191683    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:25:33.748998 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:33.748963    2565 generic.go:356] "Generic (PLEG): container finished" podID="83b19412-877c-4abe-8a05-4cafed8b6b8b" containerID="bac9cec80de267d15c38c2a4edd46c5192c43f9512ae955b455df09fadaa7132" exitCode=7
Jul 17 22:25:33.749485 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:33.749003    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm" event={"ID":"83b19412-877c-4abe-8a05-4cafed8b6b8b","Type":"ContainerDied","Data":"bac9cec80de267d15c38c2a4edd46c5192c43f9512ae955b455df09fadaa7132"}
Jul 17 22:25:33.749485 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:33.749054    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm" event={"ID":"83b19412-877c-4abe-8a05-4cafed8b6b8b","Type":"ContainerStarted","Data":"5ed608ad316de7cbd4f1b9335ddde41c256c8ec743fb24e0ef5339d9a0353e9c"}
Jul 17 22:25:33.749485 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:33.749256    2565 scope.go:117] "RemoveContainer" containerID="bac9cec80de267d15c38c2a4edd46c5192c43f9512ae955b455df09fadaa7132"
Jul 17 22:25:34.558532 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:34.558494    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584"]
Jul 17 22:25:34.558891 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:34.558879    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a66e6bad-988d-4b38-9b17-f830d5cb462d" containerName="revoke-keys"
Jul 17 22:25:34.558934 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:34.558893    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a66e6bad-988d-4b38-9b17-f830d5cb462d" containerName="revoke-keys"
Jul 17 22:25:34.558968 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:34.558957    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a66e6bad-988d-4b38-9b17-f830d5cb462d" containerName="revoke-keys"
Jul 17 22:25:34.561921 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:34.561904    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584"
Jul 17 22:25:34.569137 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:34.569101    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584"]
Jul 17 22:25:34.656326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:34.656220    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddf168f-4115-4034-81f9-31a29c6f1f4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-8w584\" (UID: \"6ddf168f-4115-4034-81f9-31a29c6f1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584"
Jul 17 22:25:34.756418 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:34.756370    2565 generic.go:356] "Generic (PLEG): container finished" podID="83b19412-877c-4abe-8a05-4cafed8b6b8b" containerID="431b35ace02084e6132cfd1e4ebe2f204c0bf92f8ce1ff9d63f9aeae6f94fea6" exitCode=7
Jul 17 22:25:34.756882 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:34.756439    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm" event={"ID":"83b19412-877c-4abe-8a05-4cafed8b6b8b","Type":"ContainerDied","Data":"431b35ace02084e6132cfd1e4ebe2f204c0bf92f8ce1ff9d63f9aeae6f94fea6"}
Jul 17 22:25:34.756882 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:34.756487    2565 scope.go:117] "RemoveContainer" containerID="bac9cec80de267d15c38c2a4edd46c5192c43f9512ae955b455df09fadaa7132"
Jul 17 22:25:34.756882 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:34.756715    2565 scope.go:117] "RemoveContainer" containerID="431b35ace02084e6132cfd1e4ebe2f204c0bf92f8ce1ff9d63f9aeae6f94fea6"
Jul 17 22:25:34.757109 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:25:34.756953    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm_odh-ai-gateway-infra(83b19412-877c-4abe-8a05-4cafed8b6b8b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm" podUID="83b19412-877c-4abe-8a05-4cafed8b6b8b"
Jul 17 22:25:34.757109 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:34.756973    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddf168f-4115-4034-81f9-31a29c6f1f4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-8w584\" (UID: \"6ddf168f-4115-4034-81f9-31a29c6f1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584"
Jul 17 22:25:34.757657 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:34.757640    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddf168f-4115-4034-81f9-31a29c6f1f4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-8w584\" (UID: \"6ddf168f-4115-4034-81f9-31a29c6f1f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584"
Jul 17 22:25:34.873902 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:34.873864    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584"
Jul 17 22:25:35.011696 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:35.011664    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584"]
Jul 17 22:25:35.013879 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:25:35.013851    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddf168f_4115_4034_81f9_31a29c6f1f4f.slice/crio-e557c95d054f6631f9e0cfbc0d699cd4eead0599dcbc23feef1b692282a3fb81 WatchSource:0}: Error finding container e557c95d054f6631f9e0cfbc0d699cd4eead0599dcbc23feef1b692282a3fb81: Status 404 returned error can't find the container with id e557c95d054f6631f9e0cfbc0d699cd4eead0599dcbc23feef1b692282a3fb81
Jul 17 22:25:35.762436 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:35.762410    2565 scope.go:117] "RemoveContainer" containerID="431b35ace02084e6132cfd1e4ebe2f204c0bf92f8ce1ff9d63f9aeae6f94fea6"
Jul 17 22:25:35.762872 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:25:35.762652    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm_odh-ai-gateway-infra(83b19412-877c-4abe-8a05-4cafed8b6b8b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm" podUID="83b19412-877c-4abe-8a05-4cafed8b6b8b"
Jul 17 22:25:35.763679 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:35.763648    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ddf168f-4115-4034-81f9-31a29c6f1f4f" containerID="c7fa76addfa1ad54ae1697110ead6d88007ea834e671f6f80685f2beeed551cc" exitCode=6
Jul 17 22:25:35.763783 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:35.763734    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584" event={"ID":"6ddf168f-4115-4034-81f9-31a29c6f1f4f","Type":"ContainerDied","Data":"c7fa76addfa1ad54ae1697110ead6d88007ea834e671f6f80685f2beeed551cc"}
Jul 17 22:25:35.763821 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:35.763791    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584" event={"ID":"6ddf168f-4115-4034-81f9-31a29c6f1f4f","Type":"ContainerStarted","Data":"e557c95d054f6631f9e0cfbc0d699cd4eead0599dcbc23feef1b692282a3fb81"}
Jul 17 22:25:35.763910 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:35.763881    2565 scope.go:117] "RemoveContainer" containerID="c7fa76addfa1ad54ae1697110ead6d88007ea834e671f6f80685f2beeed551cc"
Jul 17 22:25:36.770744 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:36.770713    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ddf168f-4115-4034-81f9-31a29c6f1f4f" containerID="4926d37f8c8c6319f44b8608b04a8ab8955d75fb030bb8d06274fe8918fa780f" exitCode=6
Jul 17 22:25:36.771255 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:36.770792    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584" event={"ID":"6ddf168f-4115-4034-81f9-31a29c6f1f4f","Type":"ContainerDied","Data":"4926d37f8c8c6319f44b8608b04a8ab8955d75fb030bb8d06274fe8918fa780f"}
Jul 17 22:25:36.771255 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:36.770846    2565 scope.go:117] "RemoveContainer" containerID="c7fa76addfa1ad54ae1697110ead6d88007ea834e671f6f80685f2beeed551cc"
Jul 17 22:25:36.771255 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:36.771104    2565 scope.go:117] "RemoveContainer" containerID="4926d37f8c8c6319f44b8608b04a8ab8955d75fb030bb8d06274fe8918fa780f"
Jul 17 22:25:36.771416 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:25:36.771361    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-8w584_odh-ai-gateway-infra(6ddf168f-4115-4034-81f9-31a29c6f1f4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584" podUID="6ddf168f-4115-4034-81f9-31a29c6f1f4f"
Jul 17 22:25:37.268900 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:37.268861    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:25:37.276565 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:37.276543    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:25:37.777294 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:37.777260    2565 scope.go:117] "RemoveContainer" containerID="4926d37f8c8c6319f44b8608b04a8ab8955d75fb030bb8d06274fe8918fa780f"
Jul 17 22:25:37.777676 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:25:37.777477    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-8w584_odh-ai-gateway-infra(6ddf168f-4115-4034-81f9-31a29c6f1f4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584" podUID="6ddf168f-4115-4034-81f9-31a29c6f1f4f"
Jul 17 22:25:48.177245 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:48.177215    2565 scope.go:117] "RemoveContainer" containerID="431b35ace02084e6132cfd1e4ebe2f204c0bf92f8ce1ff9d63f9aeae6f94fea6"
Jul 17 22:25:48.831191 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:48.831146    2565 generic.go:356] "Generic (PLEG): container finished" podID="83b19412-877c-4abe-8a05-4cafed8b6b8b" containerID="0d0b409126d11af6e1fa5bedf19f2d8d960ab83ac9c3091c96eeafae31f3e1d3" exitCode=7
Jul 17 22:25:48.831355 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:48.831220    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm" event={"ID":"83b19412-877c-4abe-8a05-4cafed8b6b8b","Type":"ContainerDied","Data":"0d0b409126d11af6e1fa5bedf19f2d8d960ab83ac9c3091c96eeafae31f3e1d3"}
Jul 17 22:25:48.831355 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:48.831305    2565 scope.go:117] "RemoveContainer" containerID="431b35ace02084e6132cfd1e4ebe2f204c0bf92f8ce1ff9d63f9aeae6f94fea6"
Jul 17 22:25:48.831674 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:48.831657    2565 scope.go:117] "RemoveContainer" containerID="0d0b409126d11af6e1fa5bedf19f2d8d960ab83ac9c3091c96eeafae31f3e1d3"
Jul 17 22:25:48.831915 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:25:48.831896    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm_odh-ai-gateway-infra(83b19412-877c-4abe-8a05-4cafed8b6b8b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm" podUID="83b19412-877c-4abe-8a05-4cafed8b6b8b"
Jul 17 22:25:49.862608 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:49.862567    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm"]
Jul 17 22:25:50.002772 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:50.002747    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm"
Jul 17 22:25:50.105162 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:50.105127    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b19412-877c-4abe-8a05-4cafed8b6b8b-maas-api-service-ca\") pod \"83b19412-877c-4abe-8a05-4cafed8b6b8b\" (UID: \"83b19412-877c-4abe-8a05-4cafed8b6b8b\") "
Jul 17 22:25:50.105534 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:50.105507    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b19412-877c-4abe-8a05-4cafed8b6b8b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83b19412-877c-4abe-8a05-4cafed8b6b8b" (UID: "83b19412-877c-4abe-8a05-4cafed8b6b8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:25:50.206823 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:50.206713    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b19412-877c-4abe-8a05-4cafed8b6b8b-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:25:50.843013 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:50.842983    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm"
Jul 17 22:25:50.843235 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:50.843031    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm" event={"ID":"83b19412-877c-4abe-8a05-4cafed8b6b8b","Type":"ContainerDied","Data":"5ed608ad316de7cbd4f1b9335ddde41c256c8ec743fb24e0ef5339d9a0353e9c"}
Jul 17 22:25:50.843235 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:50.843122    2565 scope.go:117] "RemoveContainer" containerID="0d0b409126d11af6e1fa5bedf19f2d8d960ab83ac9c3091c96eeafae31f3e1d3"
Jul 17 22:25:50.864242 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:50.864206    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm"]
Jul 17 22:25:50.867818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:50.867787    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-tn7zm"]
Jul 17 22:25:51.182826 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:51.182742    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83b19412-877c-4abe-8a05-4cafed8b6b8b" path="/var/lib/kubelet/pods/83b19412-877c-4abe-8a05-4cafed8b6b8b/volumes"
Jul 17 22:25:53.176895 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:53.176862    2565 scope.go:117] "RemoveContainer" containerID="4926d37f8c8c6319f44b8608b04a8ab8955d75fb030bb8d06274fe8918fa780f"
Jul 17 22:25:53.862546 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:53.862512    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ddf168f-4115-4034-81f9-31a29c6f1f4f" containerID="1686a07bf448dc42d68d2591abb20db6b8ea01b249c8641a9743a9ce64c9c027" exitCode=6
Jul 17 22:25:53.862731 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:53.862573    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584" event={"ID":"6ddf168f-4115-4034-81f9-31a29c6f1f4f","Type":"ContainerDied","Data":"1686a07bf448dc42d68d2591abb20db6b8ea01b249c8641a9743a9ce64c9c027"}
Jul 17 22:25:53.862731 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:53.862620    2565 scope.go:117] "RemoveContainer" containerID="4926d37f8c8c6319f44b8608b04a8ab8955d75fb030bb8d06274fe8918fa780f"
Jul 17 22:25:53.862929 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:53.862908    2565 scope.go:117] "RemoveContainer" containerID="1686a07bf448dc42d68d2591abb20db6b8ea01b249c8641a9743a9ce64c9c027"
Jul 17 22:25:53.863188 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:25:53.863168    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-8w584_odh-ai-gateway-infra(6ddf168f-4115-4034-81f9-31a29c6f1f4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584" podUID="6ddf168f-4115-4034-81f9-31a29c6f1f4f"
Jul 17 22:25:54.887906 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:54.887870    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584"]
Jul 17 22:25:55.026672 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:55.026647    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584"
Jul 17 22:25:55.053686 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:55.053650    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddf168f-4115-4034-81f9-31a29c6f1f4f-maas-api-service-ca\") pod \"6ddf168f-4115-4034-81f9-31a29c6f1f4f\" (UID: \"6ddf168f-4115-4034-81f9-31a29c6f1f4f\") "
Jul 17 22:25:55.054146 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:55.054118    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ddf168f-4115-4034-81f9-31a29c6f1f4f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ddf168f-4115-4034-81f9-31a29c6f1f4f" (UID: "6ddf168f-4115-4034-81f9-31a29c6f1f4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:25:55.155373 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:55.155266    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddf168f-4115-4034-81f9-31a29c6f1f4f-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:25:55.874683 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:55.874656    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584"
Jul 17 22:25:55.874868 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:55.874708    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584" event={"ID":"6ddf168f-4115-4034-81f9-31a29c6f1f4f","Type":"ContainerDied","Data":"e557c95d054f6631f9e0cfbc0d699cd4eead0599dcbc23feef1b692282a3fb81"}
Jul 17 22:25:55.874868 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:55.874756    2565 scope.go:117] "RemoveContainer" containerID="1686a07bf448dc42d68d2591abb20db6b8ea01b249c8641a9743a9ce64c9c027"
Jul 17 22:25:55.890145 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:55.890113    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584"]
Jul 17 22:25:55.895014 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:55.894988    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8w584"]
Jul 17 22:25:57.182625 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.182592    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ddf168f-4115-4034-81f9-31a29c6f1f4f" path="/var/lib/kubelet/pods/6ddf168f-4115-4034-81f9-31a29c6f1f4f/volumes"
Jul 17 22:25:57.890847 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.890816    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x"]
Jul 17 22:25:57.891218 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.891193    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83b19412-877c-4abe-8a05-4cafed8b6b8b" containerName="revoke-keys"
Jul 17 22:25:57.891218 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.891216    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="83b19412-877c-4abe-8a05-4cafed8b6b8b" containerName="revoke-keys"
Jul 17 22:25:57.891382 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.891233    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83b19412-877c-4abe-8a05-4cafed8b6b8b" containerName="revoke-keys"
Jul 17 22:25:57.891382 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.891239    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="83b19412-877c-4abe-8a05-4cafed8b6b8b" containerName="revoke-keys"
Jul 17 22:25:57.891382 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.891245    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ddf168f-4115-4034-81f9-31a29c6f1f4f" containerName="revoke-keys"
Jul 17 22:25:57.891382 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.891251    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ddf168f-4115-4034-81f9-31a29c6f1f4f" containerName="revoke-keys"
Jul 17 22:25:57.891382 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.891257    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ddf168f-4115-4034-81f9-31a29c6f1f4f" containerName="revoke-keys"
Jul 17 22:25:57.891382 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.891262    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ddf168f-4115-4034-81f9-31a29c6f1f4f" containerName="revoke-keys"
Jul 17 22:25:57.891382 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.891273    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83b19412-877c-4abe-8a05-4cafed8b6b8b" containerName="revoke-keys"
Jul 17 22:25:57.891382 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.891278    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="83b19412-877c-4abe-8a05-4cafed8b6b8b" containerName="revoke-keys"
Jul 17 22:25:57.891382 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.891337    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ddf168f-4115-4034-81f9-31a29c6f1f4f" containerName="revoke-keys"
Jul 17 22:25:57.891382 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.891344    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="83b19412-877c-4abe-8a05-4cafed8b6b8b" containerName="revoke-keys"
Jul 17 22:25:57.891382 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.891350    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ddf168f-4115-4034-81f9-31a29c6f1f4f" containerName="revoke-keys"
Jul 17 22:25:57.891382 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.891356    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="83b19412-877c-4abe-8a05-4cafed8b6b8b" containerName="revoke-keys"
Jul 17 22:25:57.891382 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.891361    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="83b19412-877c-4abe-8a05-4cafed8b6b8b" containerName="revoke-keys"
Jul 17 22:25:57.895660 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.895639    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x"
Jul 17 22:25:57.898938 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.898846    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:25:57.898938 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.898846    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:25:57.900744 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.900718    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x"]
Jul 17 22:25:57.982206 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:57.982110    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c31784-7864-42f4-9050-30c479ce4f74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x\" (UID: \"f3c31784-7864-42f4-9050-30c479ce4f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x"
Jul 17 22:25:58.083307 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:58.083266    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c31784-7864-42f4-9050-30c479ce4f74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x\" (UID: \"f3c31784-7864-42f4-9050-30c479ce4f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x"
Jul 17 22:25:58.083899 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:58.083878    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c31784-7864-42f4-9050-30c479ce4f74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x\" (UID: \"f3c31784-7864-42f4-9050-30c479ce4f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x"
Jul 17 22:25:58.209803 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:58.209714    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x"
Jul 17 22:25:58.347495 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:58.347469    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x"]
Jul 17 22:25:58.349431 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:25:58.349402    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c31784_7864_42f4_9050_30c479ce4f74.slice/crio-d36aac1b7be90f883c4eedc23e79743d96afb1c6a97411f824755a78a59c8308 WatchSource:0}: Error finding container d36aac1b7be90f883c4eedc23e79743d96afb1c6a97411f824755a78a59c8308: Status 404 returned error can't find the container with id d36aac1b7be90f883c4eedc23e79743d96afb1c6a97411f824755a78a59c8308
Jul 17 22:25:58.892547 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:58.892451    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x" event={"ID":"f3c31784-7864-42f4-9050-30c479ce4f74","Type":"ContainerStarted","Data":"066e318dc1d0c489812b17ec6414c3b06b2ca868e83fd4d16cd19c5705f89579"}
Jul 17 22:25:58.892547 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:58.892502    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x" event={"ID":"f3c31784-7864-42f4-9050-30c479ce4f74","Type":"ContainerStarted","Data":"d36aac1b7be90f883c4eedc23e79743d96afb1c6a97411f824755a78a59c8308"}
Jul 17 22:25:58.910181 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:58.910125    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x" podStartSLOduration=1.91010659 podStartE2EDuration="1.91010659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:25:58.908506947 +0000 UTC m=+2422.297523620" watchObservedRunningTime="2026-07-17 22:25:58.91010659 +0000 UTC m=+2422.299123231"
Jul 17 22:25:59.898963 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:59.898926    2565 generic.go:356] "Generic (PLEG): container finished" podID="f3c31784-7864-42f4-9050-30c479ce4f74" containerID="066e318dc1d0c489812b17ec6414c3b06b2ca868e83fd4d16cd19c5705f89579" exitCode=6
Jul 17 22:25:59.899349 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:59.898982    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x" event={"ID":"f3c31784-7864-42f4-9050-30c479ce4f74","Type":"ContainerDied","Data":"066e318dc1d0c489812b17ec6414c3b06b2ca868e83fd4d16cd19c5705f89579"}
Jul 17 22:25:59.899349 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:25:59.899244    2565 scope.go:117] "RemoveContainer" containerID="066e318dc1d0c489812b17ec6414c3b06b2ca868e83fd4d16cd19c5705f89579"
Jul 17 22:26:00.906491 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:00.906452    2565 generic.go:356] "Generic (PLEG): container finished" podID="f3c31784-7864-42f4-9050-30c479ce4f74" containerID="9e910d6f1947b6182b9938594141b48f29f73110abce404b8e86d34e8b6ae1c0" exitCode=6
Jul 17 22:26:00.906926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:00.906535    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x" event={"ID":"f3c31784-7864-42f4-9050-30c479ce4f74","Type":"ContainerDied","Data":"9e910d6f1947b6182b9938594141b48f29f73110abce404b8e86d34e8b6ae1c0"}
Jul 17 22:26:00.906926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:00.906590    2565 scope.go:117] "RemoveContainer" containerID="066e318dc1d0c489812b17ec6414c3b06b2ca868e83fd4d16cd19c5705f89579"
Jul 17 22:26:00.906926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:00.906777    2565 scope.go:117] "RemoveContainer" containerID="9e910d6f1947b6182b9938594141b48f29f73110abce404b8e86d34e8b6ae1c0"
Jul 17 22:26:00.907093 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:26:00.907038    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x_odh-ai-gateway-infra(f3c31784-7864-42f4-9050-30c479ce4f74)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x" podUID="f3c31784-7864-42f4-9050-30c479ce4f74"
Jul 17 22:26:01.912846 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:01.912818    2565 scope.go:117] "RemoveContainer" containerID="9e910d6f1947b6182b9938594141b48f29f73110abce404b8e86d34e8b6ae1c0"
Jul 17 22:26:01.913329 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:26:01.913097    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x_odh-ai-gateway-infra(f3c31784-7864-42f4-9050-30c479ce4f74)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x" podUID="f3c31784-7864-42f4-9050-30c479ce4f74"
Jul 17 22:26:06.108439 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:06.103823    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb"]
Jul 17 22:26:06.108439 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:06.104705    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ddf168f-4115-4034-81f9-31a29c6f1f4f" containerName="revoke-keys"
Jul 17 22:26:06.108439 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:06.104728    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ddf168f-4115-4034-81f9-31a29c6f1f4f" containerName="revoke-keys"
Jul 17 22:26:06.108439 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:06.104890    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ddf168f-4115-4034-81f9-31a29c6f1f4f" containerName="revoke-keys"
Jul 17 22:26:06.111263 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:06.111234    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb"
Jul 17 22:26:06.113404 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:06.113381    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb"]
Jul 17 22:26:06.264096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:06.264019    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad95196-e8eb-4fbc-92e7-b6d050f88ba5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-w65pb\" (UID: \"6ad95196-e8eb-4fbc-92e7-b6d050f88ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb"
Jul 17 22:26:06.365314 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:06.365218    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad95196-e8eb-4fbc-92e7-b6d050f88ba5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-w65pb\" (UID: \"6ad95196-e8eb-4fbc-92e7-b6d050f88ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb"
Jul 17 22:26:06.365825 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:06.365805    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad95196-e8eb-4fbc-92e7-b6d050f88ba5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-w65pb\" (UID: \"6ad95196-e8eb-4fbc-92e7-b6d050f88ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb"
Jul 17 22:26:06.423640 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:06.423592    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb"
Jul 17 22:26:06.560285 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:06.560258    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb"]
Jul 17 22:26:06.562635 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:26:06.562607    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad95196_e8eb_4fbc_92e7_b6d050f88ba5.slice/crio-dfc451b9e5125a24be1cd92bb756bbbdc71ec1a18620872b5ea5be535c92ac35 WatchSource:0}: Error finding container dfc451b9e5125a24be1cd92bb756bbbdc71ec1a18620872b5ea5be535c92ac35: Status 404 returned error can't find the container with id dfc451b9e5125a24be1cd92bb756bbbdc71ec1a18620872b5ea5be535c92ac35
Jul 17 22:26:06.939030 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:06.938992    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb" event={"ID":"6ad95196-e8eb-4fbc-92e7-b6d050f88ba5","Type":"ContainerStarted","Data":"5a13ad6a429d419343f8a15a4ed2e886a7434c0be52d1fbe4acea2df894969e7"}
Jul 17 22:26:06.939245 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:06.939055    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb" event={"ID":"6ad95196-e8eb-4fbc-92e7-b6d050f88ba5","Type":"ContainerStarted","Data":"dfc451b9e5125a24be1cd92bb756bbbdc71ec1a18620872b5ea5be535c92ac35"}
Jul 17 22:26:06.958581 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:06.958523    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb" podStartSLOduration=0.958505547 podStartE2EDuration="958.505547ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:26:06.956365309 +0000 UTC m=+2430.345381960" watchObservedRunningTime="2026-07-17 22:26:06.958505547 +0000 UTC m=+2430.347522176"
Jul 17 22:26:07.944823 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:07.944785    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ad95196-e8eb-4fbc-92e7-b6d050f88ba5" containerID="5a13ad6a429d419343f8a15a4ed2e886a7434c0be52d1fbe4acea2df894969e7" exitCode=6
Jul 17 22:26:07.945314 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:07.944871    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb" event={"ID":"6ad95196-e8eb-4fbc-92e7-b6d050f88ba5","Type":"ContainerDied","Data":"5a13ad6a429d419343f8a15a4ed2e886a7434c0be52d1fbe4acea2df894969e7"}
Jul 17 22:26:07.945314 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:07.945060    2565 scope.go:117] "RemoveContainer" containerID="5a13ad6a429d419343f8a15a4ed2e886a7434c0be52d1fbe4acea2df894969e7"
Jul 17 22:26:08.951479 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:08.951387    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ad95196-e8eb-4fbc-92e7-b6d050f88ba5" containerID="1fd023dfee5a7f6108a1dd55191aac858dc8e9eefdd29d0a92dd7dc41a7f4ea5" exitCode=6
Jul 17 22:26:08.951479 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:08.951467    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb" event={"ID":"6ad95196-e8eb-4fbc-92e7-b6d050f88ba5","Type":"ContainerDied","Data":"1fd023dfee5a7f6108a1dd55191aac858dc8e9eefdd29d0a92dd7dc41a7f4ea5"}
Jul 17 22:26:08.951907 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:08.951528    2565 scope.go:117] "RemoveContainer" containerID="5a13ad6a429d419343f8a15a4ed2e886a7434c0be52d1fbe4acea2df894969e7"
Jul 17 22:26:08.951907 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:08.951765    2565 scope.go:117] "RemoveContainer" containerID="1fd023dfee5a7f6108a1dd55191aac858dc8e9eefdd29d0a92dd7dc41a7f4ea5"
Jul 17 22:26:08.952009 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:26:08.951989    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-w65pb_odh-ai-gateway-infra(6ad95196-e8eb-4fbc-92e7-b6d050f88ba5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb" podUID="6ad95196-e8eb-4fbc-92e7-b6d050f88ba5"
Jul 17 22:26:09.958315 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:09.958287    2565 scope.go:117] "RemoveContainer" containerID="1fd023dfee5a7f6108a1dd55191aac858dc8e9eefdd29d0a92dd7dc41a7f4ea5"
Jul 17 22:26:09.958728 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:26:09.958516    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-w65pb_odh-ai-gateway-infra(6ad95196-e8eb-4fbc-92e7-b6d050f88ba5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb" podUID="6ad95196-e8eb-4fbc-92e7-b6d050f88ba5"
Jul 17 22:26:14.177012 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:14.176973    2565 scope.go:117] "RemoveContainer" containerID="9e910d6f1947b6182b9938594141b48f29f73110abce404b8e86d34e8b6ae1c0"
Jul 17 22:26:14.985787 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:14.985749    2565 generic.go:356] "Generic (PLEG): container finished" podID="f3c31784-7864-42f4-9050-30c479ce4f74" containerID="3c8ec5790806b367e5348685171d7999a08abd7b34500d1446171c337a2d2973" exitCode=6
Jul 17 22:26:14.985966 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:14.985822    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x" event={"ID":"f3c31784-7864-42f4-9050-30c479ce4f74","Type":"ContainerDied","Data":"3c8ec5790806b367e5348685171d7999a08abd7b34500d1446171c337a2d2973"}
Jul 17 22:26:14.985966 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:14.985879    2565 scope.go:117] "RemoveContainer" containerID="9e910d6f1947b6182b9938594141b48f29f73110abce404b8e86d34e8b6ae1c0"
Jul 17 22:26:14.986170 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:14.986154    2565 scope.go:117] "RemoveContainer" containerID="3c8ec5790806b367e5348685171d7999a08abd7b34500d1446171c337a2d2973"
Jul 17 22:26:14.986414 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:26:14.986388    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x_odh-ai-gateway-infra(f3c31784-7864-42f4-9050-30c479ce4f74)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x" podUID="f3c31784-7864-42f4-9050-30c479ce4f74"
Jul 17 22:26:16.013832 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:16.013793    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x"]
Jul 17 22:26:16.147833 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:16.147807    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x"
Jul 17 22:26:16.259876 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:16.259843    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c31784-7864-42f4-9050-30c479ce4f74-maas-api-service-ca\") pod \"f3c31784-7864-42f4-9050-30c479ce4f74\" (UID: \"f3c31784-7864-42f4-9050-30c479ce4f74\") "
Jul 17 22:26:16.260280 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:16.260252    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c31784-7864-42f4-9050-30c479ce4f74-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3c31784-7864-42f4-9050-30c479ce4f74" (UID: "f3c31784-7864-42f4-9050-30c479ce4f74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:26:16.361598 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:16.361497    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3c31784-7864-42f4-9050-30c479ce4f74-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:26:16.998873 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:16.998839    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x"
Jul 17 22:26:16.998873 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:16.998852    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x" event={"ID":"f3c31784-7864-42f4-9050-30c479ce4f74","Type":"ContainerDied","Data":"d36aac1b7be90f883c4eedc23e79743d96afb1c6a97411f824755a78a59c8308"}
Jul 17 22:26:16.999130 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:16.998908    2565 scope.go:117] "RemoveContainer" containerID="3c8ec5790806b367e5348685171d7999a08abd7b34500d1446171c337a2d2973"
Jul 17 22:26:17.023476 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:17.023434    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x"]
Jul 17 22:26:17.026336 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:17.026309    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-dj65x"]
Jul 17 22:26:17.182287 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:17.182249    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3c31784-7864-42f4-9050-30c479ce4f74" path="/var/lib/kubelet/pods/f3c31784-7864-42f4-9050-30c479ce4f74/volumes"
Jul 17 22:26:22.502141 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.502096    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg"]
Jul 17 22:26:22.502539 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.502459    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c31784-7864-42f4-9050-30c479ce4f74" containerName="revoke-keys"
Jul 17 22:26:22.502539 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.502469    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c31784-7864-42f4-9050-30c479ce4f74" containerName="revoke-keys"
Jul 17 22:26:22.502539 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.502481    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c31784-7864-42f4-9050-30c479ce4f74" containerName="revoke-keys"
Jul 17 22:26:22.502539 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.502487    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c31784-7864-42f4-9050-30c479ce4f74" containerName="revoke-keys"
Jul 17 22:26:22.502539 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.502497    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c31784-7864-42f4-9050-30c479ce4f74" containerName="revoke-keys"
Jul 17 22:26:22.502539 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.502503    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c31784-7864-42f4-9050-30c479ce4f74" containerName="revoke-keys"
Jul 17 22:26:22.502734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.502558    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c31784-7864-42f4-9050-30c479ce4f74" containerName="revoke-keys"
Jul 17 22:26:22.502734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.502567    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c31784-7864-42f4-9050-30c479ce4f74" containerName="revoke-keys"
Jul 17 22:26:22.502734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.502573    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c31784-7864-42f4-9050-30c479ce4f74" containerName="revoke-keys"
Jul 17 22:26:22.507109 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.507089    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg"
Jul 17 22:26:22.512115 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.512086    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg"]
Jul 17 22:26:22.623135 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.623096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b05d226-4360-4133-9a78-605068e12639-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg\" (UID: \"2b05d226-4360-4133-9a78-605068e12639\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg"
Jul 17 22:26:22.724281 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.724242    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b05d226-4360-4133-9a78-605068e12639-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg\" (UID: \"2b05d226-4360-4133-9a78-605068e12639\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg"
Jul 17 22:26:22.724916 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.724889    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b05d226-4360-4133-9a78-605068e12639-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg\" (UID: \"2b05d226-4360-4133-9a78-605068e12639\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg"
Jul 17 22:26:22.819174 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.819093    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg"
Jul 17 22:26:22.958254 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:22.958093    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg"]
Jul 17 22:26:22.960512 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:26:22.960485    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b05d226_4360_4133_9a78_605068e12639.slice/crio-4a639748226c86c92ce1aa8bfbf86fe5cb963d02e02d6114b31c554938d67faa WatchSource:0}: Error finding container 4a639748226c86c92ce1aa8bfbf86fe5cb963d02e02d6114b31c554938d67faa: Status 404 returned error can't find the container with id 4a639748226c86c92ce1aa8bfbf86fe5cb963d02e02d6114b31c554938d67faa
Jul 17 22:26:23.029956 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:23.029920    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg" event={"ID":"2b05d226-4360-4133-9a78-605068e12639","Type":"ContainerStarted","Data":"4a639748226c86c92ce1aa8bfbf86fe5cb963d02e02d6114b31c554938d67faa"}
Jul 17 22:26:24.036329 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:24.036293    2565 generic.go:356] "Generic (PLEG): container finished" podID="2b05d226-4360-4133-9a78-605068e12639" containerID="15d17e14f2498828fb36fb8b15fd65f4e55283639da2ab70679a84d13cf12c87" exitCode=7
Jul 17 22:26:24.036721 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:24.036346    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg" event={"ID":"2b05d226-4360-4133-9a78-605068e12639","Type":"ContainerDied","Data":"15d17e14f2498828fb36fb8b15fd65f4e55283639da2ab70679a84d13cf12c87"}
Jul 17 22:26:24.036721 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:24.036543    2565 scope.go:117] "RemoveContainer" containerID="15d17e14f2498828fb36fb8b15fd65f4e55283639da2ab70679a84d13cf12c87"
Jul 17 22:26:24.177291 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:24.177261    2565 scope.go:117] "RemoveContainer" containerID="1fd023dfee5a7f6108a1dd55191aac858dc8e9eefdd29d0a92dd7dc41a7f4ea5"
Jul 17 22:26:25.043604 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:25.043514    2565 generic.go:356] "Generic (PLEG): container finished" podID="2b05d226-4360-4133-9a78-605068e12639" containerID="b46d38296cbb9ca7e69fbdc2837adcc80a8e598451371202cd2d7ba295669aef" exitCode=7
Jul 17 22:26:25.044031 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:25.043599    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg" event={"ID":"2b05d226-4360-4133-9a78-605068e12639","Type":"ContainerDied","Data":"b46d38296cbb9ca7e69fbdc2837adcc80a8e598451371202cd2d7ba295669aef"}
Jul 17 22:26:25.044031 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:25.043657    2565 scope.go:117] "RemoveContainer" containerID="15d17e14f2498828fb36fb8b15fd65f4e55283639da2ab70679a84d13cf12c87"
Jul 17 22:26:25.044031 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:25.043917    2565 scope.go:117] "RemoveContainer" containerID="b46d38296cbb9ca7e69fbdc2837adcc80a8e598451371202cd2d7ba295669aef"
Jul 17 22:26:25.044217 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:26:25.044193    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg_odh-ai-gateway-infra(2b05d226-4360-4133-9a78-605068e12639)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg" podUID="2b05d226-4360-4133-9a78-605068e12639"
Jul 17 22:26:25.045745 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:25.045722    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ad95196-e8eb-4fbc-92e7-b6d050f88ba5" containerID="0f287e1c5c684d9ba7edfb6970d85db6c7f429b7d1b49948810da590eec45569" exitCode=6
Jul 17 22:26:25.045860 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:25.045760    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb" event={"ID":"6ad95196-e8eb-4fbc-92e7-b6d050f88ba5","Type":"ContainerDied","Data":"0f287e1c5c684d9ba7edfb6970d85db6c7f429b7d1b49948810da590eec45569"}
Jul 17 22:26:25.046036 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:25.046015    2565 scope.go:117] "RemoveContainer" containerID="0f287e1c5c684d9ba7edfb6970d85db6c7f429b7d1b49948810da590eec45569"
Jul 17 22:26:25.046207 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:26:25.046191    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-w65pb_odh-ai-gateway-infra(6ad95196-e8eb-4fbc-92e7-b6d050f88ba5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb" podUID="6ad95196-e8eb-4fbc-92e7-b6d050f88ba5"
Jul 17 22:26:25.054790 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:25.054773    2565 scope.go:117] "RemoveContainer" containerID="1fd023dfee5a7f6108a1dd55191aac858dc8e9eefdd29d0a92dd7dc41a7f4ea5"
Jul 17 22:26:26.056909 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:26.056880    2565 scope.go:117] "RemoveContainer" containerID="b46d38296cbb9ca7e69fbdc2837adcc80a8e598451371202cd2d7ba295669aef"
Jul 17 22:26:26.057446 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:26:26.057133    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg_odh-ai-gateway-infra(2b05d226-4360-4133-9a78-605068e12639)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg" podUID="2b05d226-4360-4133-9a78-605068e12639"
Jul 17 22:26:26.084527 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:26.084491    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb"]
Jul 17 22:26:26.228665 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:26.228638    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb"
Jul 17 22:26:26.356818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:26.356729    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad95196-e8eb-4fbc-92e7-b6d050f88ba5-maas-api-service-ca\") pod \"6ad95196-e8eb-4fbc-92e7-b6d050f88ba5\" (UID: \"6ad95196-e8eb-4fbc-92e7-b6d050f88ba5\") "
Jul 17 22:26:26.357199 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:26.357049    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad95196-e8eb-4fbc-92e7-b6d050f88ba5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ad95196-e8eb-4fbc-92e7-b6d050f88ba5" (UID: "6ad95196-e8eb-4fbc-92e7-b6d050f88ba5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:26:26.458195 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:26.458158    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad95196-e8eb-4fbc-92e7-b6d050f88ba5-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:26:27.064848 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:27.064814    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb" event={"ID":"6ad95196-e8eb-4fbc-92e7-b6d050f88ba5","Type":"ContainerDied","Data":"dfc451b9e5125a24be1cd92bb756bbbdc71ec1a18620872b5ea5be535c92ac35"}
Jul 17 22:26:27.064848 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:27.064841    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb"
Jul 17 22:26:27.065337 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:27.064875    2565 scope.go:117] "RemoveContainer" containerID="0f287e1c5c684d9ba7edfb6970d85db6c7f429b7d1b49948810da590eec45569"
Jul 17 22:26:27.086203 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:27.086169    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb"]
Jul 17 22:26:27.087913 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:27.087889    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-w65pb"]
Jul 17 22:26:27.182635 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:27.182600    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ad95196-e8eb-4fbc-92e7-b6d050f88ba5" path="/var/lib/kubelet/pods/6ad95196-e8eb-4fbc-92e7-b6d050f88ba5/volumes"
Jul 17 22:26:34.604727 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:34.604690    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8"]
Jul 17 22:26:34.605218 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:34.605045    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad95196-e8eb-4fbc-92e7-b6d050f88ba5" containerName="revoke-keys"
Jul 17 22:26:34.605218 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:34.605056    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad95196-e8eb-4fbc-92e7-b6d050f88ba5" containerName="revoke-keys"
Jul 17 22:26:34.605218 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:34.605090    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad95196-e8eb-4fbc-92e7-b6d050f88ba5" containerName="revoke-keys"
Jul 17 22:26:34.605218 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:34.605096    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad95196-e8eb-4fbc-92e7-b6d050f88ba5" containerName="revoke-keys"
Jul 17 22:26:34.605218 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:34.605151    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad95196-e8eb-4fbc-92e7-b6d050f88ba5" containerName="revoke-keys"
Jul 17 22:26:34.605218 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:34.605160    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad95196-e8eb-4fbc-92e7-b6d050f88ba5" containerName="revoke-keys"
Jul 17 22:26:34.605218 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:34.605169    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad95196-e8eb-4fbc-92e7-b6d050f88ba5" containerName="revoke-keys"
Jul 17 22:26:34.609371 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:34.609352    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8"
Jul 17 22:26:34.615276 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:34.615242    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8"]
Jul 17 22:26:34.633027 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:34.632989    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12f0550-4f6a-4943-bffe-efeabd9f3b0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8\" (UID: \"d12f0550-4f6a-4943-bffe-efeabd9f3b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8"
Jul 17 22:26:34.734373 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:34.734333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12f0550-4f6a-4943-bffe-efeabd9f3b0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8\" (UID: \"d12f0550-4f6a-4943-bffe-efeabd9f3b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8"
Jul 17 22:26:34.735033 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:34.735004    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12f0550-4f6a-4943-bffe-efeabd9f3b0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8\" (UID: \"d12f0550-4f6a-4943-bffe-efeabd9f3b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8"
Jul 17 22:26:34.922229 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:34.922135    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8"
Jul 17 22:26:35.076489 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:35.076296    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8"]
Jul 17 22:26:35.079344 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:26:35.079316    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12f0550_4f6a_4943_bffe_efeabd9f3b0f.slice/crio-05e6202554e994c155a835692cd326ee4916afaaee09dc5e315e7cefee022e6f WatchSource:0}: Error finding container 05e6202554e994c155a835692cd326ee4916afaaee09dc5e315e7cefee022e6f: Status 404 returned error can't find the container with id 05e6202554e994c155a835692cd326ee4916afaaee09dc5e315e7cefee022e6f
Jul 17 22:26:35.111289 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:35.111258    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8" event={"ID":"d12f0550-4f6a-4943-bffe-efeabd9f3b0f","Type":"ContainerStarted","Data":"05e6202554e994c155a835692cd326ee4916afaaee09dc5e315e7cefee022e6f"}
Jul 17 22:26:36.117778 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:36.117742    2565 generic.go:356] "Generic (PLEG): container finished" podID="d12f0550-4f6a-4943-bffe-efeabd9f3b0f" containerID="6c3a78e8ffa3b1bcdf8a81b12689a44dc568912cefc3abf48d3370c805034a33" exitCode=6
Jul 17 22:26:36.118267 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:36.117828    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8" event={"ID":"d12f0550-4f6a-4943-bffe-efeabd9f3b0f","Type":"ContainerDied","Data":"6c3a78e8ffa3b1bcdf8a81b12689a44dc568912cefc3abf48d3370c805034a33"}
Jul 17 22:26:36.118267 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:36.117979    2565 scope.go:117] "RemoveContainer" containerID="6c3a78e8ffa3b1bcdf8a81b12689a44dc568912cefc3abf48d3370c805034a33"
Jul 17 22:26:37.131997 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:37.131908    2565 generic.go:356] "Generic (PLEG): container finished" podID="d12f0550-4f6a-4943-bffe-efeabd9f3b0f" containerID="5adb664260816012180765532f47ffd91ac1c9a8f90b360c02890dab78443f26" exitCode=6
Jul 17 22:26:37.132514 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:37.131996    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8" event={"ID":"d12f0550-4f6a-4943-bffe-efeabd9f3b0f","Type":"ContainerDied","Data":"5adb664260816012180765532f47ffd91ac1c9a8f90b360c02890dab78443f26"}
Jul 17 22:26:37.132514 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:37.132098    2565 scope.go:117] "RemoveContainer" containerID="6c3a78e8ffa3b1bcdf8a81b12689a44dc568912cefc3abf48d3370c805034a33"
Jul 17 22:26:37.132514 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:37.132300    2565 scope.go:117] "RemoveContainer" containerID="5adb664260816012180765532f47ffd91ac1c9a8f90b360c02890dab78443f26"
Jul 17 22:26:37.132653 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:26:37.132522    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8_odh-ai-gateway-infra(d12f0550-4f6a-4943-bffe-efeabd9f3b0f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8" podUID="d12f0550-4f6a-4943-bffe-efeabd9f3b0f"
Jul 17 22:26:38.137974 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:38.137946    2565 scope.go:117] "RemoveContainer" containerID="5adb664260816012180765532f47ffd91ac1c9a8f90b360c02890dab78443f26"
Jul 17 22:26:38.138391 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:26:38.138190    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8_odh-ai-gateway-infra(d12f0550-4f6a-4943-bffe-efeabd9f3b0f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8" podUID="d12f0550-4f6a-4943-bffe-efeabd9f3b0f"
Jul 17 22:26:38.176259 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:38.176226    2565 scope.go:117] "RemoveContainer" containerID="b46d38296cbb9ca7e69fbdc2837adcc80a8e598451371202cd2d7ba295669aef"
Jul 17 22:26:39.144181 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:39.144092    2565 generic.go:356] "Generic (PLEG): container finished" podID="2b05d226-4360-4133-9a78-605068e12639" containerID="e1b5ea9fcf85816c9340600ed2338d3561be2578c1efc8d4b8688996ec3e2033" exitCode=7
Jul 17 22:26:39.144181 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:39.144162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg" event={"ID":"2b05d226-4360-4133-9a78-605068e12639","Type":"ContainerDied","Data":"e1b5ea9fcf85816c9340600ed2338d3561be2578c1efc8d4b8688996ec3e2033"}
Jul 17 22:26:39.144632 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:39.144224    2565 scope.go:117] "RemoveContainer" containerID="b46d38296cbb9ca7e69fbdc2837adcc80a8e598451371202cd2d7ba295669aef"
Jul 17 22:26:39.144632 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:39.144522    2565 scope.go:117] "RemoveContainer" containerID="e1b5ea9fcf85816c9340600ed2338d3561be2578c1efc8d4b8688996ec3e2033"
Jul 17 22:26:39.144800 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:26:39.144780    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg_odh-ai-gateway-infra(2b05d226-4360-4133-9a78-605068e12639)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg" podUID="2b05d226-4360-4133-9a78-605068e12639"
Jul 17 22:26:40.170293 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:40.170259    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg"]
Jul 17 22:26:40.313180 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:40.313155    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg"
Jul 17 22:26:40.392717 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:40.392683    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b05d226-4360-4133-9a78-605068e12639-maas-api-service-ca\") pod \"2b05d226-4360-4133-9a78-605068e12639\" (UID: \"2b05d226-4360-4133-9a78-605068e12639\") "
Jul 17 22:26:40.393092 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:40.393045    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b05d226-4360-4133-9a78-605068e12639-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b05d226-4360-4133-9a78-605068e12639" (UID: "2b05d226-4360-4133-9a78-605068e12639"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:26:40.493531 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:40.493447    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b05d226-4360-4133-9a78-605068e12639-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:26:41.155801 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:41.155764    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg"
Jul 17 22:26:41.156012 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:41.155802    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg" event={"ID":"2b05d226-4360-4133-9a78-605068e12639","Type":"ContainerDied","Data":"4a639748226c86c92ce1aa8bfbf86fe5cb963d02e02d6114b31c554938d67faa"}
Jul 17 22:26:41.156012 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:41.155874    2565 scope.go:117] "RemoveContainer" containerID="e1b5ea9fcf85816c9340600ed2338d3561be2578c1efc8d4b8688996ec3e2033"
Jul 17 22:26:41.176607 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:41.176577    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg"]
Jul 17 22:26:41.183427 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:41.183399    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-kzmqg"]
Jul 17 22:26:42.466967 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:42.466926    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"]
Jul 17 22:26:42.467471 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:42.467216    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg" podUID="94bb42a4-ced6-4a58-9b65-a360794d4d0b" containerName="istio-proxy" containerID="cri-o://4b1af1d9e5e7ef66dc101ae2224f63f1551f0b18575a57af2109f9a8f7caabfb" gracePeriod=30
Jul 17 22:26:42.491535 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:42.491504    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:26:42.592398 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:42.592343    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:26:43.182320 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:43.182285    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b05d226-4360-4133-9a78-605068e12639" path="/var/lib/kubelet/pods/2b05d226-4360-4133-9a78-605068e12639/volumes"
Jul 17 22:26:44.712405 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:44.712371    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:26:44.761942 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:44.761904    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:26:45.436786 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:45.436751    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:26:47.723776 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.723754    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:26:47.865439 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.865407    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-workload-certs\") pod \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") "
Jul 17 22:26:47.865604 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.865486    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-podinfo\") pod \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") "
Jul 17 22:26:47.865604 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.865525    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-credential-socket\") pod \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") "
Jul 17 22:26:47.865604 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.865542    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-envoy\") pod \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") "
Jul 17 22:26:47.865604 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.865569    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istiod-ca-cert\") pod \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") "
Jul 17 22:26:47.865809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.865667    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-token\") pod \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") "
Jul 17 22:26:47.865809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.865731    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8lls\" (UniqueName: \"kubernetes.io/projected/94bb42a4-ced6-4a58-9b65-a360794d4d0b-kube-api-access-l8lls\") pod \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") "
Jul 17 22:26:47.865809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.865759    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "94bb42a4-ced6-4a58-9b65-a360794d4d0b" (UID: "94bb42a4-ced6-4a58-9b65-a360794d4d0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:26:47.865809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.865776    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-workload-socket\") pod \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") "
Jul 17 22:26:47.866019 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.865820    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "94bb42a4-ced6-4a58-9b65-a360794d4d0b" (UID: "94bb42a4-ced6-4a58-9b65-a360794d4d0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:26:47.866019 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.865859    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-data\") pod \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\" (UID: \"94bb42a4-ced6-4a58-9b65-a360794d4d0b\") "
Jul 17 22:26:47.866019 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.865977    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "94bb42a4-ced6-4a58-9b65-a360794d4d0b" (UID: "94bb42a4-ced6-4a58-9b65-a360794d4d0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:26:47.866221 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.866095    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "94bb42a4-ced6-4a58-9b65-a360794d4d0b" (UID: "94bb42a4-ced6-4a58-9b65-a360794d4d0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:26:47.866323 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.866307    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-credential-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:26:47.866383 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.866330    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istiod-ca-cert\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:26:47.866383 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.866346    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-workload-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:26:47.866383 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.866359    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-workload-certs\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:26:47.866546 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.866514    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-data" (OuterVolumeSpecName: "istio-data") pod "94bb42a4-ced6-4a58-9b65-a360794d4d0b" (UID: "94bb42a4-ced6-4a58-9b65-a360794d4d0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:26:47.867986 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.867958    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "94bb42a4-ced6-4a58-9b65-a360794d4d0b" (UID: "94bb42a4-ced6-4a58-9b65-a360794d4d0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:26:47.867986 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.867979    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "94bb42a4-ced6-4a58-9b65-a360794d4d0b" (UID: "94bb42a4-ced6-4a58-9b65-a360794d4d0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:26:47.868264 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.868249    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-token" (OuterVolumeSpecName: "istio-token") pod "94bb42a4-ced6-4a58-9b65-a360794d4d0b" (UID: "94bb42a4-ced6-4a58-9b65-a360794d4d0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:26:47.868490 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.868467    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bb42a4-ced6-4a58-9b65-a360794d4d0b-kube-api-access-l8lls" (OuterVolumeSpecName: "kube-api-access-l8lls") pod "94bb42a4-ced6-4a58-9b65-a360794d4d0b" (UID: "94bb42a4-ced6-4a58-9b65-a360794d4d0b"). InnerVolumeSpecName "kube-api-access-l8lls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:26:47.931852 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.931810    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6"]
Jul 17 22:26:47.932274 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.932245    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b05d226-4360-4133-9a78-605068e12639" containerName="revoke-keys"
Jul 17 22:26:47.932274 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.932276    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b05d226-4360-4133-9a78-605068e12639" containerName="revoke-keys"
Jul 17 22:26:47.932444 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.932288    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b05d226-4360-4133-9a78-605068e12639" containerName="revoke-keys"
Jul 17 22:26:47.932444 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.932297    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b05d226-4360-4133-9a78-605068e12639" containerName="revoke-keys"
Jul 17 22:26:47.932444 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.932312    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94bb42a4-ced6-4a58-9b65-a360794d4d0b" containerName="istio-proxy"
Jul 17 22:26:47.932444 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.932321    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="94bb42a4-ced6-4a58-9b65-a360794d4d0b" containerName="istio-proxy"
Jul 17 22:26:47.932444 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.932331    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad95196-e8eb-4fbc-92e7-b6d050f88ba5" containerName="revoke-keys"
Jul 17 22:26:47.932444 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.932339    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad95196-e8eb-4fbc-92e7-b6d050f88ba5" containerName="revoke-keys"
Jul 17 22:26:47.932444 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.932347    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b05d226-4360-4133-9a78-605068e12639" containerName="revoke-keys"
Jul 17 22:26:47.932444 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.932355    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b05d226-4360-4133-9a78-605068e12639" containerName="revoke-keys"
Jul 17 22:26:47.932444 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.932438    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94bb42a4-ced6-4a58-9b65-a360794d4d0b" containerName="istio-proxy"
Jul 17 22:26:47.932444 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.932447    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="94bb42a4-ced6-4a58-9b65-a360794d4d0b" containerName="istio-proxy"
Jul 17 22:26:47.932922 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.932542    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b05d226-4360-4133-9a78-605068e12639" containerName="revoke-keys"
Jul 17 22:26:47.932922 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.932553    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="94bb42a4-ced6-4a58-9b65-a360794d4d0b" containerName="istio-proxy"
Jul 17 22:26:47.932922 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.932565    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="94bb42a4-ced6-4a58-9b65-a360794d4d0b" containerName="istio-proxy"
Jul 17 22:26:47.932922 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.932575    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b05d226-4360-4133-9a78-605068e12639" containerName="revoke-keys"
Jul 17 22:26:47.936009 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.935989    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6"
Jul 17 22:26:47.943823 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.943759    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6"]
Jul 17 22:26:47.966819 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.966778    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8lls\" (UniqueName: \"kubernetes.io/projected/94bb42a4-ced6-4a58-9b65-a360794d4d0b-kube-api-access-l8lls\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:26:47.966819 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.966812    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-data\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:26:47.966819 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.966824    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-podinfo\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:26:47.967049 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.966834    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-envoy\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:26:47.967049 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:47.966845    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94bb42a4-ced6-4a58-9b65-a360794d4d0b-istio-token\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:26:48.068335 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.068239    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8eb497-c9fa-453e-accb-d4f6e2c730c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6\" (UID: \"cb8eb497-c9fa-453e-accb-d4f6e2c730c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6"
Jul 17 22:26:48.169342 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.169307    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8eb497-c9fa-453e-accb-d4f6e2c730c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6\" (UID: \"cb8eb497-c9fa-453e-accb-d4f6e2c730c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6"
Jul 17 22:26:48.169950 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.169931    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8eb497-c9fa-453e-accb-d4f6e2c730c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6\" (UID: \"cb8eb497-c9fa-453e-accb-d4f6e2c730c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6"
Jul 17 22:26:48.190810 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.190774    2565 generic.go:356] "Generic (PLEG): container finished" podID="94bb42a4-ced6-4a58-9b65-a360794d4d0b" containerID="4b1af1d9e5e7ef66dc101ae2224f63f1551f0b18575a57af2109f9a8f7caabfb" exitCode=0
Jul 17 22:26:48.190976 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.190829    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"
Jul 17 22:26:48.190976 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.190865    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg" event={"ID":"94bb42a4-ced6-4a58-9b65-a360794d4d0b","Type":"ContainerDied","Data":"4b1af1d9e5e7ef66dc101ae2224f63f1551f0b18575a57af2109f9a8f7caabfb"}
Jul 17 22:26:48.190976 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.190926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg" event={"ID":"94bb42a4-ced6-4a58-9b65-a360794d4d0b","Type":"ContainerDied","Data":"1d13474ef99ba04b36c6a929ac0ada5004aadc5953d431c47f29b103374c4d47"}
Jul 17 22:26:48.190976 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.190956    2565 scope.go:117] "RemoveContainer" containerID="4b1af1d9e5e7ef66dc101ae2224f63f1551f0b18575a57af2109f9a8f7caabfb"
Jul 17 22:26:48.200397 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.200379    2565 scope.go:117] "RemoveContainer" containerID="43776e0736e8502f70f789faac0b5749ca94c57026acf97936ae4f0e8a17351a"
Jul 17 22:26:48.211032 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.211006    2565 scope.go:117] "RemoveContainer" containerID="4b1af1d9e5e7ef66dc101ae2224f63f1551f0b18575a57af2109f9a8f7caabfb"
Jul 17 22:26:48.211367 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:26:48.211347    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1af1d9e5e7ef66dc101ae2224f63f1551f0b18575a57af2109f9a8f7caabfb\": container with ID starting with 4b1af1d9e5e7ef66dc101ae2224f63f1551f0b18575a57af2109f9a8f7caabfb not found: ID does not exist" containerID="4b1af1d9e5e7ef66dc101ae2224f63f1551f0b18575a57af2109f9a8f7caabfb"
Jul 17 22:26:48.211428 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.211377    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1af1d9e5e7ef66dc101ae2224f63f1551f0b18575a57af2109f9a8f7caabfb"} err="failed to get container status \"4b1af1d9e5e7ef66dc101ae2224f63f1551f0b18575a57af2109f9a8f7caabfb\": rpc error: code = NotFound desc = could not find container \"4b1af1d9e5e7ef66dc101ae2224f63f1551f0b18575a57af2109f9a8f7caabfb\": container with ID starting with 4b1af1d9e5e7ef66dc101ae2224f63f1551f0b18575a57af2109f9a8f7caabfb not found: ID does not exist"
Jul 17 22:26:48.211428 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.211395    2565 scope.go:117] "RemoveContainer" containerID="43776e0736e8502f70f789faac0b5749ca94c57026acf97936ae4f0e8a17351a"
Jul 17 22:26:48.211726 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:26:48.211706    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43776e0736e8502f70f789faac0b5749ca94c57026acf97936ae4f0e8a17351a\": container with ID starting with 43776e0736e8502f70f789faac0b5749ca94c57026acf97936ae4f0e8a17351a not found: ID does not exist" containerID="43776e0736e8502f70f789faac0b5749ca94c57026acf97936ae4f0e8a17351a"
Jul 17 22:26:48.211779 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.211733    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43776e0736e8502f70f789faac0b5749ca94c57026acf97936ae4f0e8a17351a"} err="failed to get container status \"43776e0736e8502f70f789faac0b5749ca94c57026acf97936ae4f0e8a17351a\": rpc error: code = NotFound desc = could not find container \"43776e0736e8502f70f789faac0b5749ca94c57026acf97936ae4f0e8a17351a\": container with ID starting with 43776e0736e8502f70f789faac0b5749ca94c57026acf97936ae4f0e8a17351a not found: ID does not exist"
Jul 17 22:26:48.221729 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.221683    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"]
Jul 17 22:26:48.223026 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.222999    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ded85f30-openshift-default-7ff5575d99-csnlg"]
Jul 17 22:26:48.248234 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.248201    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6"
Jul 17 22:26:48.382805 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.382742    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6"]
Jul 17 22:26:48.385037 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:26:48.385012    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8eb497_c9fa_453e_accb_d4f6e2c730c6.slice/crio-c5fd070ef148f6b3596fe5f3ae61a643498ab1f92bb3d2b7e15d1e6d36191ce3 WatchSource:0}: Error finding container c5fd070ef148f6b3596fe5f3ae61a643498ab1f92bb3d2b7e15d1e6d36191ce3: Status 404 returned error can't find the container with id c5fd070ef148f6b3596fe5f3ae61a643498ab1f92bb3d2b7e15d1e6d36191ce3
Jul 17 22:26:48.497397 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.497365    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:26:48.667090 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.667046    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:26:48.861880 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:48.861845    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:26:49.182210 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:49.182177    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94bb42a4-ced6-4a58-9b65-a360794d4d0b" path="/var/lib/kubelet/pods/94bb42a4-ced6-4a58-9b65-a360794d4d0b/volumes"
Jul 17 22:26:49.196503 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:49.196466    2565 generic.go:356] "Generic (PLEG): container finished" podID="cb8eb497-c9fa-453e-accb-d4f6e2c730c6" containerID="6dbefc68cf683f0029beb6d402a418fb4fe83b8e61f858aefe71162e0ddbfa30" exitCode=6
Jul 17 22:26:49.196690 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:49.196549    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6" event={"ID":"cb8eb497-c9fa-453e-accb-d4f6e2c730c6","Type":"ContainerDied","Data":"6dbefc68cf683f0029beb6d402a418fb4fe83b8e61f858aefe71162e0ddbfa30"}
Jul 17 22:26:49.196690 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:49.196601    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6" event={"ID":"cb8eb497-c9fa-453e-accb-d4f6e2c730c6","Type":"ContainerStarted","Data":"c5fd070ef148f6b3596fe5f3ae61a643498ab1f92bb3d2b7e15d1e6d36191ce3"}
Jul 17 22:26:49.196809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:49.196747    2565 scope.go:117] "RemoveContainer" containerID="6dbefc68cf683f0029beb6d402a418fb4fe83b8e61f858aefe71162e0ddbfa30"
Jul 17 22:26:50.204197 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:50.204103    2565 generic.go:356] "Generic (PLEG): container finished" podID="cb8eb497-c9fa-453e-accb-d4f6e2c730c6" containerID="6dfbdf98c8440e40e0bf87fac3c7fffd7346cb2ed012bb94f3ee3a2a1ad40f1e" exitCode=6
Jul 17 22:26:50.204618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:50.204202    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6" event={"ID":"cb8eb497-c9fa-453e-accb-d4f6e2c730c6","Type":"ContainerDied","Data":"6dfbdf98c8440e40e0bf87fac3c7fffd7346cb2ed012bb94f3ee3a2a1ad40f1e"}
Jul 17 22:26:50.204618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:50.204254    2565 scope.go:117] "RemoveContainer" containerID="6dbefc68cf683f0029beb6d402a418fb4fe83b8e61f858aefe71162e0ddbfa30"
Jul 17 22:26:50.204618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:50.204415    2565 scope.go:117] "RemoveContainer" containerID="6dfbdf98c8440e40e0bf87fac3c7fffd7346cb2ed012bb94f3ee3a2a1ad40f1e"
Jul 17 22:26:50.204731 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:26:50.204656    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6_odh-ai-gateway-infra(cb8eb497-c9fa-453e-accb-d4f6e2c730c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6" podUID="cb8eb497-c9fa-453e-accb-d4f6e2c730c6"
Jul 17 22:26:51.177435 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:51.177398    2565 scope.go:117] "RemoveContainer" containerID="5adb664260816012180765532f47ffd91ac1c9a8f90b360c02890dab78443f26"
Jul 17 22:26:51.210491 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:51.210457    2565 scope.go:117] "RemoveContainer" containerID="6dfbdf98c8440e40e0bf87fac3c7fffd7346cb2ed012bb94f3ee3a2a1ad40f1e"
Jul 17 22:26:51.210838 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:26:51.210660    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6_odh-ai-gateway-infra(cb8eb497-c9fa-453e-accb-d4f6e2c730c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6" podUID="cb8eb497-c9fa-453e-accb-d4f6e2c730c6"
Jul 17 22:26:52.220513 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:52.220417    2565 generic.go:356] "Generic (PLEG): container finished" podID="d12f0550-4f6a-4943-bffe-efeabd9f3b0f" containerID="5d6bc582574ba565799c9ded4e76d4d4bdac01898a3865c2574f1c74498b8318" exitCode=6
Jul 17 22:26:52.220513 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:52.220462    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8" event={"ID":"d12f0550-4f6a-4943-bffe-efeabd9f3b0f","Type":"ContainerDied","Data":"5d6bc582574ba565799c9ded4e76d4d4bdac01898a3865c2574f1c74498b8318"}
Jul 17 22:26:52.220513 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:52.220514    2565 scope.go:117] "RemoveContainer" containerID="5adb664260816012180765532f47ffd91ac1c9a8f90b360c02890dab78443f26"
Jul 17 22:26:52.220965 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:52.220825    2565 scope.go:117] "RemoveContainer" containerID="5d6bc582574ba565799c9ded4e76d4d4bdac01898a3865c2574f1c74498b8318"
Jul 17 22:26:52.221051 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:26:52.221026    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8_odh-ai-gateway-infra(d12f0550-4f6a-4943-bffe-efeabd9f3b0f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8" podUID="d12f0550-4f6a-4943-bffe-efeabd9f3b0f"
Jul 17 22:26:53.248465 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:53.248414    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8"]
Jul 17 22:26:53.387995 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:53.387968    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8"
Jul 17 22:26:53.522231 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:53.522147    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12f0550-4f6a-4943-bffe-efeabd9f3b0f-maas-api-service-ca\") pod \"d12f0550-4f6a-4943-bffe-efeabd9f3b0f\" (UID: \"d12f0550-4f6a-4943-bffe-efeabd9f3b0f\") "
Jul 17 22:26:53.522488 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:53.522467    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12f0550-4f6a-4943-bffe-efeabd9f3b0f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d12f0550-4f6a-4943-bffe-efeabd9f3b0f" (UID: "d12f0550-4f6a-4943-bffe-efeabd9f3b0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:26:53.623700 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:53.623662    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12f0550-4f6a-4943-bffe-efeabd9f3b0f-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:26:54.164758 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:54.164726    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:26:54.230422 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:54.230396    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:26:54.233452 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:54.233432    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8"
Jul 17 22:26:54.233572 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:54.233449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8" event={"ID":"d12f0550-4f6a-4943-bffe-efeabd9f3b0f","Type":"ContainerDied","Data":"05e6202554e994c155a835692cd326ee4916afaaee09dc5e315e7cefee022e6f"}
Jul 17 22:26:54.233572 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:54.233517    2565 scope.go:117] "RemoveContainer" containerID="5d6bc582574ba565799c9ded4e76d4d4bdac01898a3865c2574f1c74498b8318"
Jul 17 22:26:54.258260 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:54.258229    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8"]
Jul 17 22:26:54.269541 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:54.269511    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-cnkt8"]
Jul 17 22:26:54.684312 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:54.684277    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:26:54.785221 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:54.785176    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:26:54.884433 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:54.884372    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:26:55.183244 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:55.183212    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d12f0550-4f6a-4943-bffe-efeabd9f3b0f" path="/var/lib/kubelet/pods/d12f0550-4f6a-4943-bffe-efeabd9f3b0f/volumes"
Jul 17 22:26:56.308226 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:26:56.308193    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:27:02.164886 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:02.164850    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:27:02.176521 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:02.176497    2565 scope.go:117] "RemoveContainer" containerID="6dfbdf98c8440e40e0bf87fac3c7fffd7346cb2ed012bb94f3ee3a2a1ad40f1e"
Jul 17 22:27:02.273799 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:02.273769    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:27:02.365689 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:02.365659    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:27:02.465709 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:02.465684    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:27:02.888862 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:02.888825    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:27:03.279061 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:03.279024    2565 generic.go:356] "Generic (PLEG): container finished" podID="cb8eb497-c9fa-453e-accb-d4f6e2c730c6" containerID="638547c7c56a5e08a3f1406c936b299d78507d683ace588e6663012b80bce09e" exitCode=6
Jul 17 22:27:03.279510 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:03.279083    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6" event={"ID":"cb8eb497-c9fa-453e-accb-d4f6e2c730c6","Type":"ContainerDied","Data":"638547c7c56a5e08a3f1406c936b299d78507d683ace588e6663012b80bce09e"}
Jul 17 22:27:03.279510 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:03.279139    2565 scope.go:117] "RemoveContainer" containerID="6dfbdf98c8440e40e0bf87fac3c7fffd7346cb2ed012bb94f3ee3a2a1ad40f1e"
Jul 17 22:27:03.279510 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:03.279447    2565 scope.go:117] "RemoveContainer" containerID="638547c7c56a5e08a3f1406c936b299d78507d683ace588e6663012b80bce09e"
Jul 17 22:27:03.279693 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:27:03.279654    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6_odh-ai-gateway-infra(cb8eb497-c9fa-453e-accb-d4f6e2c730c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6" podUID="cb8eb497-c9fa-453e-accb-d4f6e2c730c6"
Jul 17 22:27:04.306903 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:04.306866    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6"]
Jul 17 22:27:04.446684 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:04.446651    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6"
Jul 17 22:27:04.526307 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:04.526267    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8eb497-c9fa-453e-accb-d4f6e2c730c6-maas-api-service-ca\") pod \"cb8eb497-c9fa-453e-accb-d4f6e2c730c6\" (UID: \"cb8eb497-c9fa-453e-accb-d4f6e2c730c6\") "
Jul 17 22:27:04.526628 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:04.526603    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8eb497-c9fa-453e-accb-d4f6e2c730c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cb8eb497-c9fa-453e-accb-d4f6e2c730c6" (UID: "cb8eb497-c9fa-453e-accb-d4f6e2c730c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:27:04.626906 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:04.626806    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8eb497-c9fa-453e-accb-d4f6e2c730c6-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:27:05.291593 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:05.291556    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6" event={"ID":"cb8eb497-c9fa-453e-accb-d4f6e2c730c6","Type":"ContainerDied","Data":"c5fd070ef148f6b3596fe5f3ae61a643498ab1f92bb3d2b7e15d1e6d36191ce3"}
Jul 17 22:27:05.291593 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:05.291591    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6"
Jul 17 22:27:05.291871 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:05.291626    2565 scope.go:117] "RemoveContainer" containerID="638547c7c56a5e08a3f1406c936b299d78507d683ace588e6663012b80bce09e"
Jul 17 22:27:05.308963 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:05.308921    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6"]
Jul 17 22:27:05.312682 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:05.312656    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nr5z6"]
Jul 17 22:27:06.181604 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.181575    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q"]
Jul 17 22:27:06.181933 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.181919    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12f0550-4f6a-4943-bffe-efeabd9f3b0f" containerName="revoke-keys"
Jul 17 22:27:06.181990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.181937    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12f0550-4f6a-4943-bffe-efeabd9f3b0f" containerName="revoke-keys"
Jul 17 22:27:06.181990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.181950    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb8eb497-c9fa-453e-accb-d4f6e2c730c6" containerName="revoke-keys"
Jul 17 22:27:06.181990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.181956    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb8eb497-c9fa-453e-accb-d4f6e2c730c6" containerName="revoke-keys"
Jul 17 22:27:06.181990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.181964    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb8eb497-c9fa-453e-accb-d4f6e2c730c6" containerName="revoke-keys"
Jul 17 22:27:06.181990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.181969    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb8eb497-c9fa-453e-accb-d4f6e2c730c6" containerName="revoke-keys"
Jul 17 22:27:06.181990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.181981    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12f0550-4f6a-4943-bffe-efeabd9f3b0f" containerName="revoke-keys"
Jul 17 22:27:06.181990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.181986    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12f0550-4f6a-4943-bffe-efeabd9f3b0f" containerName="revoke-keys"
Jul 17 22:27:06.181990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.181991    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12f0550-4f6a-4943-bffe-efeabd9f3b0f" containerName="revoke-keys"
Jul 17 22:27:06.182318 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.181996    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12f0550-4f6a-4943-bffe-efeabd9f3b0f" containerName="revoke-keys"
Jul 17 22:27:06.182318 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.182080    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b05d226-4360-4133-9a78-605068e12639" containerName="revoke-keys"
Jul 17 22:27:06.182318 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.182093    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12f0550-4f6a-4943-bffe-efeabd9f3b0f" containerName="revoke-keys"
Jul 17 22:27:06.182318 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.182099    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb8eb497-c9fa-453e-accb-d4f6e2c730c6" containerName="revoke-keys"
Jul 17 22:27:06.182318 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.182107    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb8eb497-c9fa-453e-accb-d4f6e2c730c6" containerName="revoke-keys"
Jul 17 22:27:06.182318 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.182112    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12f0550-4f6a-4943-bffe-efeabd9f3b0f" containerName="revoke-keys"
Jul 17 22:27:06.182318 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.182117    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb8eb497-c9fa-453e-accb-d4f6e2c730c6" containerName="revoke-keys"
Jul 17 22:27:06.186378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.186359    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q"
Jul 17 22:27:06.190582 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.190562    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:27:06.190711 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.190585    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:27:06.204974 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.204942    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q"]
Jul 17 22:27:06.241041 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.241001    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a6c2ef-78be-43d7-8cbe-c6a930a61a3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-b664q\" (UID: \"14a6c2ef-78be-43d7-8cbe-c6a930a61a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q"
Jul 17 22:27:06.342673 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.342635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a6c2ef-78be-43d7-8cbe-c6a930a61a3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-b664q\" (UID: \"14a6c2ef-78be-43d7-8cbe-c6a930a61a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q"
Jul 17 22:27:06.343425 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.343398    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a6c2ef-78be-43d7-8cbe-c6a930a61a3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-b664q\" (UID: \"14a6c2ef-78be-43d7-8cbe-c6a930a61a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q"
Jul 17 22:27:06.496230 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.496199    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q"
Jul 17 22:27:06.631474 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:06.631436    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q"]
Jul 17 22:27:06.634483 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:27:06.634450    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a6c2ef_78be_43d7_8cbe_c6a930a61a3a.slice/crio-90195a453ceaee688502acd85a7c34f5087965663d0beb48f366953210df035c WatchSource:0}: Error finding container 90195a453ceaee688502acd85a7c34f5087965663d0beb48f366953210df035c: Status 404 returned error can't find the container with id 90195a453ceaee688502acd85a7c34f5087965663d0beb48f366953210df035c
Jul 17 22:27:07.182978 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:07.182902    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb8eb497-c9fa-453e-accb-d4f6e2c730c6" path="/var/lib/kubelet/pods/cb8eb497-c9fa-453e-accb-d4f6e2c730c6/volumes"
Jul 17 22:27:07.304914 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:07.304879    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q" event={"ID":"14a6c2ef-78be-43d7-8cbe-c6a930a61a3a","Type":"ContainerStarted","Data":"36722b9947442cc0ebfdd7675d15f39f88ff558c1f94d78af27dd62acaea80c2"}
Jul 17 22:27:07.305111 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:07.304931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q" event={"ID":"14a6c2ef-78be-43d7-8cbe-c6a930a61a3a","Type":"ContainerStarted","Data":"90195a453ceaee688502acd85a7c34f5087965663d0beb48f366953210df035c"}
Jul 17 22:27:07.331701 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:07.331649    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q" podStartSLOduration=1.331634121 podStartE2EDuration="1.331634121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:27:07.330218597 +0000 UTC m=+2490.719235250" watchObservedRunningTime="2026-07-17 22:27:07.331634121 +0000 UTC m=+2490.720650771"
Jul 17 22:27:08.311512 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:08.311477    2565 generic.go:356] "Generic (PLEG): container finished" podID="14a6c2ef-78be-43d7-8cbe-c6a930a61a3a" containerID="36722b9947442cc0ebfdd7675d15f39f88ff558c1f94d78af27dd62acaea80c2" exitCode=6
Jul 17 22:27:08.312009 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:08.311564    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q" event={"ID":"14a6c2ef-78be-43d7-8cbe-c6a930a61a3a","Type":"ContainerDied","Data":"36722b9947442cc0ebfdd7675d15f39f88ff558c1f94d78af27dd62acaea80c2"}
Jul 17 22:27:08.312009 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:08.311726    2565 scope.go:117] "RemoveContainer" containerID="36722b9947442cc0ebfdd7675d15f39f88ff558c1f94d78af27dd62acaea80c2"
Jul 17 22:27:09.317775 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:09.317741    2565 generic.go:356] "Generic (PLEG): container finished" podID="14a6c2ef-78be-43d7-8cbe-c6a930a61a3a" containerID="3f649f8a8bf324a694ac269a2f5a61548972d3e5a074f44757bf7d960b6cc9fd" exitCode=6
Jul 17 22:27:09.318212 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:09.317822    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q" event={"ID":"14a6c2ef-78be-43d7-8cbe-c6a930a61a3a","Type":"ContainerDied","Data":"3f649f8a8bf324a694ac269a2f5a61548972d3e5a074f44757bf7d960b6cc9fd"}
Jul 17 22:27:09.318212 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:09.317888    2565 scope.go:117] "RemoveContainer" containerID="36722b9947442cc0ebfdd7675d15f39f88ff558c1f94d78af27dd62acaea80c2"
Jul 17 22:27:09.318212 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:09.318109    2565 scope.go:117] "RemoveContainer" containerID="3f649f8a8bf324a694ac269a2f5a61548972d3e5a074f44757bf7d960b6cc9fd"
Jul 17 22:27:09.318362 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:27:09.318344    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-b664q_odh-ai-gateway-infra(14a6c2ef-78be-43d7-8cbe-c6a930a61a3a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q" podUID="14a6c2ef-78be-43d7-8cbe-c6a930a61a3a"
Jul 17 22:27:10.324385 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:10.324355    2565 scope.go:117] "RemoveContainer" containerID="3f649f8a8bf324a694ac269a2f5a61548972d3e5a074f44757bf7d960b6cc9fd"
Jul 17 22:27:10.324783 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:27:10.324578    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-b664q_odh-ai-gateway-infra(14a6c2ef-78be-43d7-8cbe-c6a930a61a3a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q" podUID="14a6c2ef-78be-43d7-8cbe-c6a930a61a3a"
Jul 17 22:27:12.539896 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:12.539855    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj"]
Jul 17 22:27:12.540454 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:12.540433    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb8eb497-c9fa-453e-accb-d4f6e2c730c6" containerName="revoke-keys"
Jul 17 22:27:12.540524 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:12.540457    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb8eb497-c9fa-453e-accb-d4f6e2c730c6" containerName="revoke-keys"
Jul 17 22:27:12.540581 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:12.540573    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12f0550-4f6a-4943-bffe-efeabd9f3b0f" containerName="revoke-keys"
Jul 17 22:27:12.543466 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:12.543444    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj"
Jul 17 22:27:12.549795 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:12.549764    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj"]
Jul 17 22:27:12.598046 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:12.598005    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/035473aa-a9e2-414e-8d13-a2ad1a7354ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj\" (UID: \"035473aa-a9e2-414e-8d13-a2ad1a7354ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj"
Jul 17 22:27:12.698858 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:12.698817    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/035473aa-a9e2-414e-8d13-a2ad1a7354ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj\" (UID: \"035473aa-a9e2-414e-8d13-a2ad1a7354ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj"
Jul 17 22:27:12.699492 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:12.699471    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/035473aa-a9e2-414e-8d13-a2ad1a7354ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj\" (UID: \"035473aa-a9e2-414e-8d13-a2ad1a7354ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj"
Jul 17 22:27:12.855095 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:12.854970    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj"
Jul 17 22:27:12.991177 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:12.991151    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj"]
Jul 17 22:27:12.993350 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:27:12.993320    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod035473aa_a9e2_414e_8d13_a2ad1a7354ed.slice/crio-9f35b05445b18c426fc0b50b01798c3861776b630528a7a73ae4b6499038d1b3 WatchSource:0}: Error finding container 9f35b05445b18c426fc0b50b01798c3861776b630528a7a73ae4b6499038d1b3: Status 404 returned error can't find the container with id 9f35b05445b18c426fc0b50b01798c3861776b630528a7a73ae4b6499038d1b3
Jul 17 22:27:13.339163 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:13.339124    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj" event={"ID":"035473aa-a9e2-414e-8d13-a2ad1a7354ed","Type":"ContainerStarted","Data":"3d26f4e25dd8544258fb24f26aa1874ced455a7b71fc45326062e8a1e437e937"}
Jul 17 22:27:13.339336 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:13.339180    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj" event={"ID":"035473aa-a9e2-414e-8d13-a2ad1a7354ed","Type":"ContainerStarted","Data":"9f35b05445b18c426fc0b50b01798c3861776b630528a7a73ae4b6499038d1b3"}
Jul 17 22:27:13.357793 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:13.357739    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj" podStartSLOduration=1.357722119 podStartE2EDuration="1.357722119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:27:13.356127153 +0000 UTC m=+2496.745143800" watchObservedRunningTime="2026-07-17 22:27:13.357722119 +0000 UTC m=+2496.746738769"
Jul 17 22:27:14.345671 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:14.345635    2565 generic.go:356] "Generic (PLEG): container finished" podID="035473aa-a9e2-414e-8d13-a2ad1a7354ed" containerID="3d26f4e25dd8544258fb24f26aa1874ced455a7b71fc45326062e8a1e437e937" exitCode=6
Jul 17 22:27:14.346103 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:14.345732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj" event={"ID":"035473aa-a9e2-414e-8d13-a2ad1a7354ed","Type":"ContainerDied","Data":"3d26f4e25dd8544258fb24f26aa1874ced455a7b71fc45326062e8a1e437e937"}
Jul 17 22:27:14.346103 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:14.345886    2565 scope.go:117] "RemoveContainer" containerID="3d26f4e25dd8544258fb24f26aa1874ced455a7b71fc45326062e8a1e437e937"
Jul 17 22:27:15.352502 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:15.352467    2565 generic.go:356] "Generic (PLEG): container finished" podID="035473aa-a9e2-414e-8d13-a2ad1a7354ed" containerID="73909cb0eda79def4a98a5278b1cd6885a7a612d2336b297532e694dfffb1dfa" exitCode=6
Jul 17 22:27:15.352906 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:15.352530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj" event={"ID":"035473aa-a9e2-414e-8d13-a2ad1a7354ed","Type":"ContainerDied","Data":"73909cb0eda79def4a98a5278b1cd6885a7a612d2336b297532e694dfffb1dfa"}
Jul 17 22:27:15.352906 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:15.352593    2565 scope.go:117] "RemoveContainer" containerID="3d26f4e25dd8544258fb24f26aa1874ced455a7b71fc45326062e8a1e437e937"
Jul 17 22:27:15.352906 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:15.352769    2565 scope.go:117] "RemoveContainer" containerID="73909cb0eda79def4a98a5278b1cd6885a7a612d2336b297532e694dfffb1dfa"
Jul 17 22:27:15.353115 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:27:15.353020    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj_odh-ai-gateway-infra(035473aa-a9e2-414e-8d13-a2ad1a7354ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj" podUID="035473aa-a9e2-414e-8d13-a2ad1a7354ed"
Jul 17 22:27:16.359231 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:16.359203    2565 scope.go:117] "RemoveContainer" containerID="73909cb0eda79def4a98a5278b1cd6885a7a612d2336b297532e694dfffb1dfa"
Jul 17 22:27:16.359620 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:27:16.359408    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj_odh-ai-gateway-infra(035473aa-a9e2-414e-8d13-a2ad1a7354ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj" podUID="035473aa-a9e2-414e-8d13-a2ad1a7354ed"
Jul 17 22:27:22.176920 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:22.176889    2565 scope.go:117] "RemoveContainer" containerID="3f649f8a8bf324a694ac269a2f5a61548972d3e5a074f44757bf7d960b6cc9fd"
Jul 17 22:27:23.395569 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:23.395529    2565 generic.go:356] "Generic (PLEG): container finished" podID="14a6c2ef-78be-43d7-8cbe-c6a930a61a3a" containerID="69c4320fc24d8dadf9c6b3f62f92c5adf210a0626b868647321b7c244826cb81" exitCode=6
Jul 17 22:27:23.396047 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:23.395599    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q" event={"ID":"14a6c2ef-78be-43d7-8cbe-c6a930a61a3a","Type":"ContainerDied","Data":"69c4320fc24d8dadf9c6b3f62f92c5adf210a0626b868647321b7c244826cb81"}
Jul 17 22:27:23.396047 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:23.395658    2565 scope.go:117] "RemoveContainer" containerID="3f649f8a8bf324a694ac269a2f5a61548972d3e5a074f44757bf7d960b6cc9fd"
Jul 17 22:27:23.396047 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:23.396026    2565 scope.go:117] "RemoveContainer" containerID="69c4320fc24d8dadf9c6b3f62f92c5adf210a0626b868647321b7c244826cb81"
Jul 17 22:27:23.396327 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:27:23.396306    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-b664q_odh-ai-gateway-infra(14a6c2ef-78be-43d7-8cbe-c6a930a61a3a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q" podUID="14a6c2ef-78be-43d7-8cbe-c6a930a61a3a"
Jul 17 22:27:24.425625 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.425589    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q"]
Jul 17 22:27:24.568860 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.568834    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q"
Jul 17 22:27:24.607629 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.607599    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a6c2ef-78be-43d7-8cbe-c6a930a61a3a-maas-api-service-ca\") pod \"14a6c2ef-78be-43d7-8cbe-c6a930a61a3a\" (UID: \"14a6c2ef-78be-43d7-8cbe-c6a930a61a3a\") "
Jul 17 22:27:24.607977 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.607953    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a6c2ef-78be-43d7-8cbe-c6a930a61a3a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14a6c2ef-78be-43d7-8cbe-c6a930a61a3a" (UID: "14a6c2ef-78be-43d7-8cbe-c6a930a61a3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:27:24.642318 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.642288    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl"]
Jul 17 22:27:24.642649 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.642636    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a6c2ef-78be-43d7-8cbe-c6a930a61a3a" containerName="revoke-keys"
Jul 17 22:27:24.642693 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.642651    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a6c2ef-78be-43d7-8cbe-c6a930a61a3a" containerName="revoke-keys"
Jul 17 22:27:24.642693 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.642664    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a6c2ef-78be-43d7-8cbe-c6a930a61a3a" containerName="revoke-keys"
Jul 17 22:27:24.642693 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.642669    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a6c2ef-78be-43d7-8cbe-c6a930a61a3a" containerName="revoke-keys"
Jul 17 22:27:24.642693 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.642679    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a6c2ef-78be-43d7-8cbe-c6a930a61a3a" containerName="revoke-keys"
Jul 17 22:27:24.642693 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.642684    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a6c2ef-78be-43d7-8cbe-c6a930a61a3a" containerName="revoke-keys"
Jul 17 22:27:24.642848 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.642740    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="14a6c2ef-78be-43d7-8cbe-c6a930a61a3a" containerName="revoke-keys"
Jul 17 22:27:24.642848 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.642750    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="14a6c2ef-78be-43d7-8cbe-c6a930a61a3a" containerName="revoke-keys"
Jul 17 22:27:24.645806 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.645791    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl"
Jul 17 22:27:24.653060 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.653038    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl"]
Jul 17 22:27:24.709376 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.709279    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74483d6-57ac-4417-b729-f9521ceb14cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl\" (UID: \"f74483d6-57ac-4417-b729-f9521ceb14cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl"
Jul 17 22:27:24.709530 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.709426    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14a6c2ef-78be-43d7-8cbe-c6a930a61a3a-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:27:24.810237 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.810198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74483d6-57ac-4417-b729-f9521ceb14cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl\" (UID: \"f74483d6-57ac-4417-b729-f9521ceb14cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl"
Jul 17 22:27:24.810853 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.810826    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74483d6-57ac-4417-b729-f9521ceb14cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl\" (UID: \"f74483d6-57ac-4417-b729-f9521ceb14cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl"
Jul 17 22:27:24.957490 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:24.957460    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl"
Jul 17 22:27:25.092821 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:25.092794    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl"]
Jul 17 22:27:25.095104 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:27:25.095075    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74483d6_57ac_4417_b729_f9521ceb14cf.slice/crio-9e0fa9020113f7c6d149c976cefb58cc6ffcb1df500395f95ac02d533e753dd0 WatchSource:0}: Error finding container 9e0fa9020113f7c6d149c976cefb58cc6ffcb1df500395f95ac02d533e753dd0: Status 404 returned error can't find the container with id 9e0fa9020113f7c6d149c976cefb58cc6ffcb1df500395f95ac02d533e753dd0
Jul 17 22:27:25.407973 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:25.407942    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q"
Jul 17 22:27:25.408208 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:25.407941    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q" event={"ID":"14a6c2ef-78be-43d7-8cbe-c6a930a61a3a","Type":"ContainerDied","Data":"90195a453ceaee688502acd85a7c34f5087965663d0beb48f366953210df035c"}
Jul 17 22:27:25.408208 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:25.408135    2565 scope.go:117] "RemoveContainer" containerID="69c4320fc24d8dadf9c6b3f62f92c5adf210a0626b868647321b7c244826cb81"
Jul 17 22:27:25.409675 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:25.409646    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl" event={"ID":"f74483d6-57ac-4417-b729-f9521ceb14cf","Type":"ContainerStarted","Data":"06f6eef579c4b7bb482e82062db7144f196c50364315878d86a7650728670511"}
Jul 17 22:27:25.409784 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:25.409699    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl" event={"ID":"f74483d6-57ac-4417-b729-f9521ceb14cf","Type":"ContainerStarted","Data":"9e0fa9020113f7c6d149c976cefb58cc6ffcb1df500395f95ac02d533e753dd0"}
Jul 17 22:27:25.426405 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:25.426357    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl" podStartSLOduration=1.426341365 podStartE2EDuration="1.426341365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:27:25.426283788 +0000 UTC m=+2508.815300467" watchObservedRunningTime="2026-07-17 22:27:25.426341365 +0000 UTC m=+2508.815358013"
Jul 17 22:27:25.439214 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:25.439175    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q"]
Jul 17 22:27:25.442995 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:25.442971    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-b664q"]
Jul 17 22:27:26.416881 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:26.416851    2565 generic.go:356] "Generic (PLEG): container finished" podID="f74483d6-57ac-4417-b729-f9521ceb14cf" containerID="06f6eef579c4b7bb482e82062db7144f196c50364315878d86a7650728670511" exitCode=6
Jul 17 22:27:26.417042 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:26.416944    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl" event={"ID":"f74483d6-57ac-4417-b729-f9521ceb14cf","Type":"ContainerDied","Data":"06f6eef579c4b7bb482e82062db7144f196c50364315878d86a7650728670511"}
Jul 17 22:27:26.417132 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:26.417116    2565 scope.go:117] "RemoveContainer" containerID="06f6eef579c4b7bb482e82062db7144f196c50364315878d86a7650728670511"
Jul 17 22:27:27.184633 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:27.184597    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14a6c2ef-78be-43d7-8cbe-c6a930a61a3a" path="/var/lib/kubelet/pods/14a6c2ef-78be-43d7-8cbe-c6a930a61a3a/volumes"
Jul 17 22:27:27.423739 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:27.423699    2565 generic.go:356] "Generic (PLEG): container finished" podID="f74483d6-57ac-4417-b729-f9521ceb14cf" containerID="91919839a90cc4b5d61cc8b60792ef592a70739cb272cc5428f9430849f24c6f" exitCode=6
Jul 17 22:27:27.423927 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:27.423788    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl" event={"ID":"f74483d6-57ac-4417-b729-f9521ceb14cf","Type":"ContainerDied","Data":"91919839a90cc4b5d61cc8b60792ef592a70739cb272cc5428f9430849f24c6f"}
Jul 17 22:27:27.423927 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:27.423848    2565 scope.go:117] "RemoveContainer" containerID="06f6eef579c4b7bb482e82062db7144f196c50364315878d86a7650728670511"
Jul 17 22:27:27.424109 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:27.424093    2565 scope.go:117] "RemoveContainer" containerID="91919839a90cc4b5d61cc8b60792ef592a70739cb272cc5428f9430849f24c6f"
Jul 17 22:27:27.424361 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:27:27.424343    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl_odh-ai-gateway-infra(f74483d6-57ac-4417-b729-f9521ceb14cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl" podUID="f74483d6-57ac-4417-b729-f9521ceb14cf"
Jul 17 22:27:28.431726 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:28.431653    2565 scope.go:117] "RemoveContainer" containerID="91919839a90cc4b5d61cc8b60792ef592a70739cb272cc5428f9430849f24c6f"
Jul 17 22:27:28.432123 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:27:28.431905    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl_odh-ai-gateway-infra(f74483d6-57ac-4417-b729-f9521ceb14cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl" podUID="f74483d6-57ac-4417-b729-f9521ceb14cf"
Jul 17 22:27:28.828305 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:28.828266    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:27:30.176600 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:30.176571    2565 scope.go:117] "RemoveContainer" containerID="73909cb0eda79def4a98a5278b1cd6885a7a612d2336b297532e694dfffb1dfa"
Jul 17 22:27:31.448420 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:31.448388    2565 generic.go:356] "Generic (PLEG): container finished" podID="035473aa-a9e2-414e-8d13-a2ad1a7354ed" containerID="e468900c97317559690e3d43cccb98b1700557f33e1e72fd377dc5d89a93bfa1" exitCode=6
Jul 17 22:27:31.448856 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:31.448455    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj" event={"ID":"035473aa-a9e2-414e-8d13-a2ad1a7354ed","Type":"ContainerDied","Data":"e468900c97317559690e3d43cccb98b1700557f33e1e72fd377dc5d89a93bfa1"}
Jul 17 22:27:31.448856 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:31.448520    2565 scope.go:117] "RemoveContainer" containerID="73909cb0eda79def4a98a5278b1cd6885a7a612d2336b297532e694dfffb1dfa"
Jul 17 22:27:31.448856 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:31.448805    2565 scope.go:117] "RemoveContainer" containerID="e468900c97317559690e3d43cccb98b1700557f33e1e72fd377dc5d89a93bfa1"
Jul 17 22:27:31.449043 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:27:31.449024    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj_odh-ai-gateway-infra(035473aa-a9e2-414e-8d13-a2ad1a7354ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj" podUID="035473aa-a9e2-414e-8d13-a2ad1a7354ed"
Jul 17 22:27:32.473083 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:32.473028    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj"]
Jul 17 22:27:32.609295 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:32.609270    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj"
Jul 17 22:27:32.685338 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:32.685295    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/035473aa-a9e2-414e-8d13-a2ad1a7354ed-maas-api-service-ca\") pod \"035473aa-a9e2-414e-8d13-a2ad1a7354ed\" (UID: \"035473aa-a9e2-414e-8d13-a2ad1a7354ed\") "
Jul 17 22:27:32.685679 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:32.685656    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/035473aa-a9e2-414e-8d13-a2ad1a7354ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "035473aa-a9e2-414e-8d13-a2ad1a7354ed" (UID: "035473aa-a9e2-414e-8d13-a2ad1a7354ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:27:32.786381 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:32.786337    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/035473aa-a9e2-414e-8d13-a2ad1a7354ed-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:27:33.460965 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:33.460879    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj"
Jul 17 22:27:33.460965 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:33.460887    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj" event={"ID":"035473aa-a9e2-414e-8d13-a2ad1a7354ed","Type":"ContainerDied","Data":"9f35b05445b18c426fc0b50b01798c3861776b630528a7a73ae4b6499038d1b3"}
Jul 17 22:27:33.460965 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:33.460947    2565 scope.go:117] "RemoveContainer" containerID="e468900c97317559690e3d43cccb98b1700557f33e1e72fd377dc5d89a93bfa1"
Jul 17 22:27:33.476115 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:33.476086    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj"]
Jul 17 22:27:33.482419 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:33.482395    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-ccdhj"]
Jul 17 22:27:34.739532 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:34.739496    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:27:34.843342 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:34.843301    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:27:34.940152 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:34.940093    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:27:35.182002 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:35.181969    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="035473aa-a9e2-414e-8d13-a2ad1a7354ed" path="/var/lib/kubelet/pods/035473aa-a9e2-414e-8d13-a2ad1a7354ed/volumes"
Jul 17 22:27:37.973903 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:37.973822    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv"]
Jul 17 22:27:37.974312 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:37.974259    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="035473aa-a9e2-414e-8d13-a2ad1a7354ed" containerName="revoke-keys"
Jul 17 22:27:37.974312 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:37.974274    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="035473aa-a9e2-414e-8d13-a2ad1a7354ed" containerName="revoke-keys"
Jul 17 22:27:37.974312 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:37.974286    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="035473aa-a9e2-414e-8d13-a2ad1a7354ed" containerName="revoke-keys"
Jul 17 22:27:37.974312 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:37.974291    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="035473aa-a9e2-414e-8d13-a2ad1a7354ed" containerName="revoke-keys"
Jul 17 22:27:37.974457 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:37.974360    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="035473aa-a9e2-414e-8d13-a2ad1a7354ed" containerName="revoke-keys"
Jul 17 22:27:37.974457 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:37.974367    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="035473aa-a9e2-414e-8d13-a2ad1a7354ed" containerName="revoke-keys"
Jul 17 22:27:37.974457 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:37.974374    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="14a6c2ef-78be-43d7-8cbe-c6a930a61a3a" containerName="revoke-keys"
Jul 17 22:27:37.978836 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:37.978816    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv"
Jul 17 22:27:37.983551 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:37.983525    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv"]
Jul 17 22:27:38.135018 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:38.134986    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0996374-7ab1-4120-b71c-d00f21697349-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv\" (UID: \"c0996374-7ab1-4120-b71c-d00f21697349\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv"
Jul 17 22:27:38.236324 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:38.236232    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0996374-7ab1-4120-b71c-d00f21697349-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv\" (UID: \"c0996374-7ab1-4120-b71c-d00f21697349\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv"
Jul 17 22:27:38.236867 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:38.236847    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0996374-7ab1-4120-b71c-d00f21697349-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv\" (UID: \"c0996374-7ab1-4120-b71c-d00f21697349\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv"
Jul 17 22:27:38.290730 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:38.290694    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv"
Jul 17 22:27:38.452996 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:38.452968    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv"]
Jul 17 22:27:38.454972 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:27:38.454939    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0996374_7ab1_4120_b71c_d00f21697349.slice/crio-e5a5a139af6c94f187972489f68d947ce58365090bdf0db4f896650322ce64d6 WatchSource:0}: Error finding container e5a5a139af6c94f187972489f68d947ce58365090bdf0db4f896650322ce64d6: Status 404 returned error can't find the container with id e5a5a139af6c94f187972489f68d947ce58365090bdf0db4f896650322ce64d6
Jul 17 22:27:38.486824 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:38.486741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv" event={"ID":"c0996374-7ab1-4120-b71c-d00f21697349","Type":"ContainerStarted","Data":"e5a5a139af6c94f187972489f68d947ce58365090bdf0db4f896650322ce64d6"}
Jul 17 22:27:39.492711 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:39.492628    2565 generic.go:356] "Generic (PLEG): container finished" podID="c0996374-7ab1-4120-b71c-d00f21697349" containerID="9d41eef7023f6f7da647fc3ac84c79d06f5921e52db6989726a1f6cc5170bc24" exitCode=6
Jul 17 22:27:39.493274 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:39.492722    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv" event={"ID":"c0996374-7ab1-4120-b71c-d00f21697349","Type":"ContainerDied","Data":"9d41eef7023f6f7da647fc3ac84c79d06f5921e52db6989726a1f6cc5170bc24"}
Jul 17 22:27:39.493274 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:39.492877    2565 scope.go:117] "RemoveContainer" containerID="9d41eef7023f6f7da647fc3ac84c79d06f5921e52db6989726a1f6cc5170bc24"
Jul 17 22:27:40.500835 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:40.500802    2565 generic.go:356] "Generic (PLEG): container finished" podID="c0996374-7ab1-4120-b71c-d00f21697349" containerID="0743a96e63b1047564b3352012edf17497fdbb009c0333ca0dd6a4bddcb8b4c1" exitCode=6
Jul 17 22:27:40.501402 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:40.500850    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv" event={"ID":"c0996374-7ab1-4120-b71c-d00f21697349","Type":"ContainerDied","Data":"0743a96e63b1047564b3352012edf17497fdbb009c0333ca0dd6a4bddcb8b4c1"}
Jul 17 22:27:40.501402 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:40.500897    2565 scope.go:117] "RemoveContainer" containerID="9d41eef7023f6f7da647fc3ac84c79d06f5921e52db6989726a1f6cc5170bc24"
Jul 17 22:27:40.501402 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:40.501164    2565 scope.go:117] "RemoveContainer" containerID="0743a96e63b1047564b3352012edf17497fdbb009c0333ca0dd6a4bddcb8b4c1"
Jul 17 22:27:40.501575 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:27:40.501417    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv_odh-ai-gateway-infra(c0996374-7ab1-4120-b71c-d00f21697349)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv" podUID="c0996374-7ab1-4120-b71c-d00f21697349"
Jul 17 22:27:41.507609 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:41.507575    2565 scope.go:117] "RemoveContainer" containerID="0743a96e63b1047564b3352012edf17497fdbb009c0333ca0dd6a4bddcb8b4c1"
Jul 17 22:27:41.508003 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:27:41.507799    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv_odh-ai-gateway-infra(c0996374-7ab1-4120-b71c-d00f21697349)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv" podUID="c0996374-7ab1-4120-b71c-d00f21697349"
Jul 17 22:27:42.176800 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:42.176760    2565 scope.go:117] "RemoveContainer" containerID="91919839a90cc4b5d61cc8b60792ef592a70739cb272cc5428f9430849f24c6f"
Jul 17 22:27:42.513443 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:42.513410    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl" event={"ID":"f74483d6-57ac-4417-b729-f9521ceb14cf","Type":"ContainerStarted","Data":"9b1cbc68d995e6c179fb5a3cd73b25ccc128903d78b4eab6882081aae8c5a6e5"}
Jul 17 22:27:43.519246 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:43.519212    2565 generic.go:356] "Generic (PLEG): container finished" podID="f74483d6-57ac-4417-b729-f9521ceb14cf" containerID="9b1cbc68d995e6c179fb5a3cd73b25ccc128903d78b4eab6882081aae8c5a6e5" exitCode=6
Jul 17 22:27:43.519664 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:43.519263    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl" event={"ID":"f74483d6-57ac-4417-b729-f9521ceb14cf","Type":"ContainerDied","Data":"9b1cbc68d995e6c179fb5a3cd73b25ccc128903d78b4eab6882081aae8c5a6e5"}
Jul 17 22:27:43.519664 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:43.519321    2565 scope.go:117] "RemoveContainer" containerID="91919839a90cc4b5d61cc8b60792ef592a70739cb272cc5428f9430849f24c6f"
Jul 17 22:27:43.519664 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:43.519592    2565 scope.go:117] "RemoveContainer" containerID="9b1cbc68d995e6c179fb5a3cd73b25ccc128903d78b4eab6882081aae8c5a6e5"
Jul 17 22:27:43.519833 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:27:43.519814    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl_odh-ai-gateway-infra(f74483d6-57ac-4417-b729-f9521ceb14cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl" podUID="f74483d6-57ac-4417-b729-f9521ceb14cf"
Jul 17 22:27:43.543314 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:43.543280    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl"]
Jul 17 22:27:44.669377 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:44.669351    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl"
Jul 17 22:27:44.798001 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:44.797908    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74483d6-57ac-4417-b729-f9521ceb14cf-maas-api-service-ca\") pod \"f74483d6-57ac-4417-b729-f9521ceb14cf\" (UID: \"f74483d6-57ac-4417-b729-f9521ceb14cf\") "
Jul 17 22:27:44.798307 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:44.798280    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74483d6-57ac-4417-b729-f9521ceb14cf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f74483d6-57ac-4417-b729-f9521ceb14cf" (UID: "f74483d6-57ac-4417-b729-f9521ceb14cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:27:44.899176 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:44.899146    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74483d6-57ac-4417-b729-f9521ceb14cf-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:27:45.531406 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:45.531375    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl"
Jul 17 22:27:45.531618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:45.531367    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl" event={"ID":"f74483d6-57ac-4417-b729-f9521ceb14cf","Type":"ContainerDied","Data":"9e0fa9020113f7c6d149c976cefb58cc6ffcb1df500395f95ac02d533e753dd0"}
Jul 17 22:27:45.531618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:45.531528    2565 scope.go:117] "RemoveContainer" containerID="9b1cbc68d995e6c179fb5a3cd73b25ccc128903d78b4eab6882081aae8c5a6e5"
Jul 17 22:27:45.547169 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:45.547138    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl"]
Jul 17 22:27:45.554131 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:45.554107    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-mkcwl"]
Jul 17 22:27:47.182401 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:47.182363    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f74483d6-57ac-4417-b729-f9521ceb14cf" path="/var/lib/kubelet/pods/f74483d6-57ac-4417-b729-f9521ceb14cf/volumes"
Jul 17 22:27:50.824475 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:50.824440    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:27:53.176862 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:53.176826    2565 scope.go:117] "RemoveContainer" containerID="0743a96e63b1047564b3352012edf17497fdbb009c0333ca0dd6a4bddcb8b4c1"
Jul 17 22:27:53.577680 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:53.577645    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv" event={"ID":"c0996374-7ab1-4120-b71c-d00f21697349","Type":"ContainerStarted","Data":"0e93b5c67096bd45ce73cce3f5270e807a06003558ac0044013de7d6a799b6ee"}
Jul 17 22:27:53.593806 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:53.593742    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv" podStartSLOduration=16.59372067 podStartE2EDuration="16.59372067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:27:53.593440986 +0000 UTC m=+2536.982457638" watchObservedRunningTime="2026-07-17 22:27:53.59372067 +0000 UTC m=+2536.982737301"
Jul 17 22:27:54.584003 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:54.583967    2565 generic.go:356] "Generic (PLEG): container finished" podID="c0996374-7ab1-4120-b71c-d00f21697349" containerID="0e93b5c67096bd45ce73cce3f5270e807a06003558ac0044013de7d6a799b6ee" exitCode=6
Jul 17 22:27:54.584463 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:54.584038    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv" event={"ID":"c0996374-7ab1-4120-b71c-d00f21697349","Type":"ContainerDied","Data":"0e93b5c67096bd45ce73cce3f5270e807a06003558ac0044013de7d6a799b6ee"}
Jul 17 22:27:54.584463 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:54.584139    2565 scope.go:117] "RemoveContainer" containerID="0743a96e63b1047564b3352012edf17497fdbb009c0333ca0dd6a4bddcb8b4c1"
Jul 17 22:27:54.584463 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:54.584419    2565 scope.go:117] "RemoveContainer" containerID="0e93b5c67096bd45ce73cce3f5270e807a06003558ac0044013de7d6a799b6ee"
Jul 17 22:27:54.584622 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:27:54.584603    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv_odh-ai-gateway-infra(c0996374-7ab1-4120-b71c-d00f21697349)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv" podUID="c0996374-7ab1-4120-b71c-d00f21697349"
Jul 17 22:27:54.609036 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:54.608997    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv"]
Jul 17 22:27:55.727807 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:55.727781    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv"
Jul 17 22:27:55.794453 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:55.794420    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0996374-7ab1-4120-b71c-d00f21697349-maas-api-service-ca\") pod \"c0996374-7ab1-4120-b71c-d00f21697349\" (UID: \"c0996374-7ab1-4120-b71c-d00f21697349\") "
Jul 17 22:27:55.794824 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:55.794798    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0996374-7ab1-4120-b71c-d00f21697349-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c0996374-7ab1-4120-b71c-d00f21697349" (UID: "c0996374-7ab1-4120-b71c-d00f21697349"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:27:55.895589 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:55.895499    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0996374-7ab1-4120-b71c-d00f21697349-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:27:56.208710 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.208629    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh"]
Jul 17 22:27:56.208980 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.208967    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f74483d6-57ac-4417-b729-f9521ceb14cf" containerName="revoke-keys"
Jul 17 22:27:56.209022 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.208981    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f74483d6-57ac-4417-b729-f9521ceb14cf" containerName="revoke-keys"
Jul 17 22:27:56.209022 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.208990    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f74483d6-57ac-4417-b729-f9521ceb14cf" containerName="revoke-keys"
Jul 17 22:27:56.209022 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.208995    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f74483d6-57ac-4417-b729-f9521ceb14cf" containerName="revoke-keys"
Jul 17 22:27:56.209022 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.209004    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0996374-7ab1-4120-b71c-d00f21697349" containerName="revoke-keys"
Jul 17 22:27:56.209022 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.209009    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0996374-7ab1-4120-b71c-d00f21697349" containerName="revoke-keys"
Jul 17 22:27:56.209192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.209026    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="035473aa-a9e2-414e-8d13-a2ad1a7354ed" containerName="revoke-keys"
Jul 17 22:27:56.209192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.209031    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="035473aa-a9e2-414e-8d13-a2ad1a7354ed" containerName="revoke-keys"
Jul 17 22:27:56.209192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.209037    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0996374-7ab1-4120-b71c-d00f21697349" containerName="revoke-keys"
Jul 17 22:27:56.209192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.209042    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0996374-7ab1-4120-b71c-d00f21697349" containerName="revoke-keys"
Jul 17 22:27:56.209192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.209051    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0996374-7ab1-4120-b71c-d00f21697349" containerName="revoke-keys"
Jul 17 22:27:56.209192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.209056    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0996374-7ab1-4120-b71c-d00f21697349" containerName="revoke-keys"
Jul 17 22:27:56.209192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.209120    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f74483d6-57ac-4417-b729-f9521ceb14cf" containerName="revoke-keys"
Jul 17 22:27:56.209192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.209128    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f74483d6-57ac-4417-b729-f9521ceb14cf" containerName="revoke-keys"
Jul 17 22:27:56.209192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.209134    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0996374-7ab1-4120-b71c-d00f21697349" containerName="revoke-keys"
Jul 17 22:27:56.209192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.209140    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0996374-7ab1-4120-b71c-d00f21697349" containerName="revoke-keys"
Jul 17 22:27:56.209192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.209147    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="035473aa-a9e2-414e-8d13-a2ad1a7354ed" containerName="revoke-keys"
Jul 17 22:27:56.212200 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.212184    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh"
Jul 17 22:27:56.220464 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.220438    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh"]
Jul 17 22:27:56.298564 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.298525    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad489fd-b38a-4479-b443-954e11ed67f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh\" (UID: \"aad489fd-b38a-4479-b443-954e11ed67f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh"
Jul 17 22:27:56.399242 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.399203    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad489fd-b38a-4479-b443-954e11ed67f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh\" (UID: \"aad489fd-b38a-4479-b443-954e11ed67f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh"
Jul 17 22:27:56.399880 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.399853    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad489fd-b38a-4479-b443-954e11ed67f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh\" (UID: \"aad489fd-b38a-4479-b443-954e11ed67f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh"
Jul 17 22:27:56.523212 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.523170    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh"
Jul 17 22:27:56.598020 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.597987    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv"
Jul 17 22:27:56.598238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.598021    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv" event={"ID":"c0996374-7ab1-4120-b71c-d00f21697349","Type":"ContainerDied","Data":"e5a5a139af6c94f187972489f68d947ce58365090bdf0db4f896650322ce64d6"}
Jul 17 22:27:56.598238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.598114    2565 scope.go:117] "RemoveContainer" containerID="0e93b5c67096bd45ce73cce3f5270e807a06003558ac0044013de7d6a799b6ee"
Jul 17 22:27:56.608122 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.608096    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:27:56.626680 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.626640    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv"]
Jul 17 22:27:56.636366 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.636328    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-ndkdv"]
Jul 17 22:27:56.668115 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.668091    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh"]
Jul 17 22:27:56.669968 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:27:56.669936    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad489fd_b38a_4479_b443_954e11ed67f6.slice/crio-fb0ac8036ead68eec15b858a69347e7ec343c7fa7a13f3ad4dfd9535de5ba6c4 WatchSource:0}: Error finding container fb0ac8036ead68eec15b858a69347e7ec343c7fa7a13f3ad4dfd9535de5ba6c4: Status 404 returned error can't find the container with id fb0ac8036ead68eec15b858a69347e7ec343c7fa7a13f3ad4dfd9535de5ba6c4
Jul 17 22:27:56.708537 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.708506    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:27:56.807910 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:56.807821    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:27:57.182881 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:57.182787    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0996374-7ab1-4120-b71c-d00f21697349" path="/var/lib/kubelet/pods/c0996374-7ab1-4120-b71c-d00f21697349/volumes"
Jul 17 22:27:57.605451 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:57.605420    2565 generic.go:356] "Generic (PLEG): container finished" podID="aad489fd-b38a-4479-b443-954e11ed67f6" containerID="f1f45bc19254acabdfa01d1cead89ee1a9b989e48bc3865abf1d7f8d1ed44033" exitCode=6
Jul 17 22:27:57.605650 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:57.605499    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh" event={"ID":"aad489fd-b38a-4479-b443-954e11ed67f6","Type":"ContainerDied","Data":"f1f45bc19254acabdfa01d1cead89ee1a9b989e48bc3865abf1d7f8d1ed44033"}
Jul 17 22:27:57.605650 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:57.605546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh" event={"ID":"aad489fd-b38a-4479-b443-954e11ed67f6","Type":"ContainerStarted","Data":"fb0ac8036ead68eec15b858a69347e7ec343c7fa7a13f3ad4dfd9535de5ba6c4"}
Jul 17 22:27:57.605780 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:57.605659    2565 scope.go:117] "RemoveContainer" containerID="f1f45bc19254acabdfa01d1cead89ee1a9b989e48bc3865abf1d7f8d1ed44033"
Jul 17 22:27:58.612460 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:58.612425    2565 generic.go:356] "Generic (PLEG): container finished" podID="aad489fd-b38a-4479-b443-954e11ed67f6" containerID="0264088bc5c454e57dbe011d36dc9aefa7d5f95ad7b65916d21770f8f20b47d2" exitCode=6
Jul 17 22:27:58.612940 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:58.612514    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh" event={"ID":"aad489fd-b38a-4479-b443-954e11ed67f6","Type":"ContainerDied","Data":"0264088bc5c454e57dbe011d36dc9aefa7d5f95ad7b65916d21770f8f20b47d2"}
Jul 17 22:27:58.612940 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:58.612577    2565 scope.go:117] "RemoveContainer" containerID="f1f45bc19254acabdfa01d1cead89ee1a9b989e48bc3865abf1d7f8d1ed44033"
Jul 17 22:27:58.612940 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:58.612806    2565 scope.go:117] "RemoveContainer" containerID="0264088bc5c454e57dbe011d36dc9aefa7d5f95ad7b65916d21770f8f20b47d2"
Jul 17 22:27:58.613130 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:27:58.613054    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh_odh-ai-gateway-infra(aad489fd-b38a-4479-b443-954e11ed67f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh" podUID="aad489fd-b38a-4479-b443-954e11ed67f6"
Jul 17 22:27:59.619138 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:27:59.619108    2565 scope.go:117] "RemoveContainer" containerID="0264088bc5c454e57dbe011d36dc9aefa7d5f95ad7b65916d21770f8f20b47d2"
Jul 17 22:27:59.619554 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:27:59.619308    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh_odh-ai-gateway-infra(aad489fd-b38a-4479-b443-954e11ed67f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh" podUID="aad489fd-b38a-4479-b443-954e11ed67f6"
Jul 17 22:28:07.312593 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:07.312548    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:08.160273 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:08.160241    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:12.583317 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:12.583279    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh"]
Jul 17 22:28:12.583785 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:12.583639    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f74483d6-57ac-4417-b729-f9521ceb14cf" containerName="revoke-keys"
Jul 17 22:28:12.583785 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:12.583651    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f74483d6-57ac-4417-b729-f9521ceb14cf" containerName="revoke-keys"
Jul 17 22:28:12.583785 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:12.583716    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f74483d6-57ac-4417-b729-f9521ceb14cf" containerName="revoke-keys"
Jul 17 22:28:12.583785 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:12.583724    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0996374-7ab1-4120-b71c-d00f21697349" containerName="revoke-keys"
Jul 17 22:28:12.587561 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:12.587542    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh"
Jul 17 22:28:12.594374 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:12.594332    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh"]
Jul 17 22:28:12.650405 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:12.650371    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6a4aeb-32b1-41e3-953a-7b1241297db0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh\" (UID: \"9f6a4aeb-32b1-41e3-953a-7b1241297db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh"
Jul 17 22:28:12.751865 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:12.751829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6a4aeb-32b1-41e3-953a-7b1241297db0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh\" (UID: \"9f6a4aeb-32b1-41e3-953a-7b1241297db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh"
Jul 17 22:28:12.752568 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:12.752548    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6a4aeb-32b1-41e3-953a-7b1241297db0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh\" (UID: \"9f6a4aeb-32b1-41e3-953a-7b1241297db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh"
Jul 17 22:28:12.898777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:12.898680    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh"
Jul 17 22:28:13.035965 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:13.035938    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh"]
Jul 17 22:28:13.038853 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:28:13.038830    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6a4aeb_32b1_41e3_953a_7b1241297db0.slice/crio-ceca7f3e22919c5fc3afad8e8ff3549c9c7568678c1083c2e3c2844c5d25e37b WatchSource:0}: Error finding container ceca7f3e22919c5fc3afad8e8ff3549c9c7568678c1083c2e3c2844c5d25e37b: Status 404 returned error can't find the container with id ceca7f3e22919c5fc3afad8e8ff3549c9c7568678c1083c2e3c2844c5d25e37b
Jul 17 22:28:13.685461 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:13.685420    2565 generic.go:356] "Generic (PLEG): container finished" podID="9f6a4aeb-32b1-41e3-953a-7b1241297db0" containerID="97c5f7150c758f27ce51bc7244f737361ebd667e0e133c2c5ecceada2ef57c51" exitCode=6
Jul 17 22:28:13.685799 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:13.685491    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh" event={"ID":"9f6a4aeb-32b1-41e3-953a-7b1241297db0","Type":"ContainerDied","Data":"97c5f7150c758f27ce51bc7244f737361ebd667e0e133c2c5ecceada2ef57c51"}
Jul 17 22:28:13.685799 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:13.685541    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh" event={"ID":"9f6a4aeb-32b1-41e3-953a-7b1241297db0","Type":"ContainerStarted","Data":"ceca7f3e22919c5fc3afad8e8ff3549c9c7568678c1083c2e3c2844c5d25e37b"}
Jul 17 22:28:13.685799 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:13.685717    2565 scope.go:117] "RemoveContainer" containerID="97c5f7150c758f27ce51bc7244f737361ebd667e0e133c2c5ecceada2ef57c51"
Jul 17 22:28:14.265773 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:14.265741    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:14.333162 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:14.333127    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:14.412764 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:14.412730    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:14.511479 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:14.511347    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:14.613242 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:14.613192    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:14.692537 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:14.692507    2565 generic.go:356] "Generic (PLEG): container finished" podID="9f6a4aeb-32b1-41e3-953a-7b1241297db0" containerID="4a21e2603d8f24f28fd36d376f93c87c2a9b520905e9e0f67a768e80407dbd6d" exitCode=6
Jul 17 22:28:14.692934 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:14.692601    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh" event={"ID":"9f6a4aeb-32b1-41e3-953a-7b1241297db0","Type":"ContainerDied","Data":"4a21e2603d8f24f28fd36d376f93c87c2a9b520905e9e0f67a768e80407dbd6d"}
Jul 17 22:28:14.692934 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:14.692672    2565 scope.go:117] "RemoveContainer" containerID="97c5f7150c758f27ce51bc7244f737361ebd667e0e133c2c5ecceada2ef57c51"
Jul 17 22:28:14.692934 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:14.692822    2565 scope.go:117] "RemoveContainer" containerID="4a21e2603d8f24f28fd36d376f93c87c2a9b520905e9e0f67a768e80407dbd6d"
Jul 17 22:28:14.693082 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:28:14.693039    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh_odh-ai-gateway-infra(9f6a4aeb-32b1-41e3-953a-7b1241297db0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh" podUID="9f6a4aeb-32b1-41e3-953a-7b1241297db0"
Jul 17 22:28:15.176637 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:15.176606    2565 scope.go:117] "RemoveContainer" containerID="0264088bc5c454e57dbe011d36dc9aefa7d5f95ad7b65916d21770f8f20b47d2"
Jul 17 22:28:15.371565 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:15.371537    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:15.458837 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:15.458808    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:15.559367 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:15.559326    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:15.699695 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:15.699660    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh" event={"ID":"aad489fd-b38a-4479-b443-954e11ed67f6","Type":"ContainerStarted","Data":"314b434fe4587f6e81093684ad860a4cd211c98444080f56374f79388580e226"}
Jul 17 22:28:15.701367 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:15.701347    2565 scope.go:117] "RemoveContainer" containerID="4a21e2603d8f24f28fd36d376f93c87c2a9b520905e9e0f67a768e80407dbd6d"
Jul 17 22:28:15.701533 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:28:15.701515    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh_odh-ai-gateway-infra(9f6a4aeb-32b1-41e3-953a-7b1241297db0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh" podUID="9f6a4aeb-32b1-41e3-953a-7b1241297db0"
Jul 17 22:28:15.716365 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:15.716260    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh" podStartSLOduration=19.716243238 podStartE2EDuration="19.716243238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:28:15.715423296 +0000 UTC m=+2559.104439937" watchObservedRunningTime="2026-07-17 22:28:15.716243238 +0000 UTC m=+2559.105259889"
Jul 17 22:28:16.707572 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:16.707540    2565 generic.go:356] "Generic (PLEG): container finished" podID="aad489fd-b38a-4479-b443-954e11ed67f6" containerID="314b434fe4587f6e81093684ad860a4cd211c98444080f56374f79388580e226" exitCode=6
Jul 17 22:28:16.708014 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:16.707614    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh" event={"ID":"aad489fd-b38a-4479-b443-954e11ed67f6","Type":"ContainerDied","Data":"314b434fe4587f6e81093684ad860a4cd211c98444080f56374f79388580e226"}
Jul 17 22:28:16.708014 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:16.707674    2565 scope.go:117] "RemoveContainer" containerID="0264088bc5c454e57dbe011d36dc9aefa7d5f95ad7b65916d21770f8f20b47d2"
Jul 17 22:28:16.708014 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:16.707888    2565 scope.go:117] "RemoveContainer" containerID="314b434fe4587f6e81093684ad860a4cd211c98444080f56374f79388580e226"
Jul 17 22:28:16.708209 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:28:16.708118    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh_odh-ai-gateway-infra(aad489fd-b38a-4479-b443-954e11ed67f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh" podUID="aad489fd-b38a-4479-b443-954e11ed67f6"
Jul 17 22:28:16.734112 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:16.734052    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh"]
Jul 17 22:28:16.743794 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:16.743763    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:28:17.249285 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:17.249248    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:17.853331 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:17.853304    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh"
Jul 17 22:28:17.999506 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:17.999477    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad489fd-b38a-4479-b443-954e11ed67f6-maas-api-service-ca\") pod \"aad489fd-b38a-4479-b443-954e11ed67f6\" (UID: \"aad489fd-b38a-4479-b443-954e11ed67f6\") "
Jul 17 22:28:17.999890 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:17.999864    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad489fd-b38a-4479-b443-954e11ed67f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aad489fd-b38a-4479-b443-954e11ed67f6" (UID: "aad489fd-b38a-4479-b443-954e11ed67f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:28:18.100723 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:18.100670    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad489fd-b38a-4479-b443-954e11ed67f6-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:28:18.719824 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:18.719787    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh"
Jul 17 22:28:18.720013 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:18.719823    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh" event={"ID":"aad489fd-b38a-4479-b443-954e11ed67f6","Type":"ContainerDied","Data":"fb0ac8036ead68eec15b858a69347e7ec343c7fa7a13f3ad4dfd9535de5ba6c4"}
Jul 17 22:28:18.720013 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:18.719908    2565 scope.go:117] "RemoveContainer" containerID="314b434fe4587f6e81093684ad860a4cd211c98444080f56374f79388580e226"
Jul 17 22:28:18.743866 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:18.743831    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh"]
Jul 17 22:28:18.750109 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:18.750058    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rvnkh"]
Jul 17 22:28:19.182262 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:19.182227    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aad489fd-b38a-4479-b443-954e11ed67f6" path="/var/lib/kubelet/pods/aad489fd-b38a-4479-b443-954e11ed67f6/volumes"
Jul 17 22:28:22.991084 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:22.991033    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:23.060179 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:23.060146    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:23.675126 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:23.675092    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:24.682683 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:24.682644    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2"]
Jul 17 22:28:24.683110 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:24.683009    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad489fd-b38a-4479-b443-954e11ed67f6" containerName="revoke-keys"
Jul 17 22:28:24.683110 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:24.683019    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad489fd-b38a-4479-b443-954e11ed67f6" containerName="revoke-keys"
Jul 17 22:28:24.683110 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:24.683028    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad489fd-b38a-4479-b443-954e11ed67f6" containerName="revoke-keys"
Jul 17 22:28:24.683110 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:24.683033    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad489fd-b38a-4479-b443-954e11ed67f6" containerName="revoke-keys"
Jul 17 22:28:24.683251 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:24.683119    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad489fd-b38a-4479-b443-954e11ed67f6" containerName="revoke-keys"
Jul 17 22:28:24.683251 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:24.683130    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad489fd-b38a-4479-b443-954e11ed67f6" containerName="revoke-keys"
Jul 17 22:28:24.683251 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:24.683136    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad489fd-b38a-4479-b443-954e11ed67f6" containerName="revoke-keys"
Jul 17 22:28:24.687687 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:24.687670    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2"
Jul 17 22:28:24.692100 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:24.692042    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2"]
Jul 17 22:28:24.763951 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:24.763913    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f14bf336-6a5a-45a9-9794-77f01de59d81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-skrb2\" (UID: \"f14bf336-6a5a-45a9-9794-77f01de59d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2"
Jul 17 22:28:24.865249 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:24.865208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f14bf336-6a5a-45a9-9794-77f01de59d81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-skrb2\" (UID: \"f14bf336-6a5a-45a9-9794-77f01de59d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2"
Jul 17 22:28:24.865851 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:24.865828    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f14bf336-6a5a-45a9-9794-77f01de59d81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-skrb2\" (UID: \"f14bf336-6a5a-45a9-9794-77f01de59d81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2"
Jul 17 22:28:24.999518 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:24.999487    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2"
Jul 17 22:28:25.133608 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:25.133582    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2"]
Jul 17 22:28:25.138444 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:28:25.138392    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14bf336_6a5a_45a9_9794_77f01de59d81.slice/crio-3bc2b46882daa1682dfbade0e23838b05fb3aef1696614ea12ddda71bc131a3b WatchSource:0}: Error finding container 3bc2b46882daa1682dfbade0e23838b05fb3aef1696614ea12ddda71bc131a3b: Status 404 returned error can't find the container with id 3bc2b46882daa1682dfbade0e23838b05fb3aef1696614ea12ddda71bc131a3b
Jul 17 22:28:25.755304 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:25.755271    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2" event={"ID":"f14bf336-6a5a-45a9-9794-77f01de59d81","Type":"ContainerStarted","Data":"e9e05eca5a94b38263700ed1fd20f5eb9768c1a7dabb248c87d19878c15a3a0f"}
Jul 17 22:28:25.755696 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:25.755324    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2" event={"ID":"f14bf336-6a5a-45a9-9794-77f01de59d81","Type":"ContainerStarted","Data":"3bc2b46882daa1682dfbade0e23838b05fb3aef1696614ea12ddda71bc131a3b"}
Jul 17 22:28:25.772759 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:25.772701    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2" podStartSLOduration=1.772679718 podStartE2EDuration="1.772679718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:28:25.770291178 +0000 UTC m=+2569.159307822" watchObservedRunningTime="2026-07-17 22:28:25.772679718 +0000 UTC m=+2569.161696370"
Jul 17 22:28:25.842870 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:25.842825    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:28:26.761317 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:26.761275    2565 generic.go:356] "Generic (PLEG): container finished" podID="f14bf336-6a5a-45a9-9794-77f01de59d81" containerID="e9e05eca5a94b38263700ed1fd20f5eb9768c1a7dabb248c87d19878c15a3a0f" exitCode=6
Jul 17 22:28:26.761317 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:26.761321    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2" event={"ID":"f14bf336-6a5a-45a9-9794-77f01de59d81","Type":"ContainerDied","Data":"e9e05eca5a94b38263700ed1fd20f5eb9768c1a7dabb248c87d19878c15a3a0f"}
Jul 17 22:28:26.761808 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:26.761535    2565 scope.go:117] "RemoveContainer" containerID="e9e05eca5a94b38263700ed1fd20f5eb9768c1a7dabb248c87d19878c15a3a0f"
Jul 17 22:28:27.768104 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:27.768075    2565 generic.go:356] "Generic (PLEG): container finished" podID="f14bf336-6a5a-45a9-9794-77f01de59d81" containerID="e9863879a63d8808afea50ed09711839dbea2a5de0797ac5e2548c8a7eb0a20d" exitCode=6
Jul 17 22:28:27.768586 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:27.768169    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2" event={"ID":"f14bf336-6a5a-45a9-9794-77f01de59d81","Type":"ContainerDied","Data":"e9863879a63d8808afea50ed09711839dbea2a5de0797ac5e2548c8a7eb0a20d"}
Jul 17 22:28:27.768586 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:27.768228    2565 scope.go:117] "RemoveContainer" containerID="e9e05eca5a94b38263700ed1fd20f5eb9768c1a7dabb248c87d19878c15a3a0f"
Jul 17 22:28:27.768586 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:27.768440    2565 scope.go:117] "RemoveContainer" containerID="e9863879a63d8808afea50ed09711839dbea2a5de0797ac5e2548c8a7eb0a20d"
Jul 17 22:28:27.768744 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:28:27.768679    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-skrb2_odh-ai-gateway-infra(f14bf336-6a5a-45a9-9794-77f01de59d81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2" podUID="f14bf336-6a5a-45a9-9794-77f01de59d81"
Jul 17 22:28:28.017180 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:28.017144    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6"]
Jul 17 22:28:28.017516 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:28.017503    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad489fd-b38a-4479-b443-954e11ed67f6" containerName="revoke-keys"
Jul 17 22:28:28.017561 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:28.017518    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad489fd-b38a-4479-b443-954e11ed67f6" containerName="revoke-keys"
Jul 17 22:28:28.021763 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:28.021720    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6"
Jul 17 22:28:28.027450 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:28.027425    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6"]
Jul 17 22:28:28.094713 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:28.094675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73da79ed-e9dd-4af5-9819-7858900fbf5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6\" (UID: \"73da79ed-e9dd-4af5-9819-7858900fbf5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6"
Jul 17 22:28:28.176521 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:28.176480    2565 scope.go:117] "RemoveContainer" containerID="4a21e2603d8f24f28fd36d376f93c87c2a9b520905e9e0f67a768e80407dbd6d"
Jul 17 22:28:28.196147 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:28.196111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73da79ed-e9dd-4af5-9819-7858900fbf5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6\" (UID: \"73da79ed-e9dd-4af5-9819-7858900fbf5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6"
Jul 17 22:28:28.196782 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:28.196752    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73da79ed-e9dd-4af5-9819-7858900fbf5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6\" (UID: \"73da79ed-e9dd-4af5-9819-7858900fbf5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6"
Jul 17 22:28:28.333537 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:28.333444    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6"
Jul 17 22:28:28.544890 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:28.544862    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6"]
Jul 17 22:28:28.547512 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:28:28.547456    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73da79ed_e9dd_4af5_9819_7858900fbf5b.slice/crio-27c1c57f1f7e81ed186b2b307caca6e49e058dcb490511a038178c1011517b63 WatchSource:0}: Error finding container 27c1c57f1f7e81ed186b2b307caca6e49e058dcb490511a038178c1011517b63: Status 404 returned error can't find the container with id 27c1c57f1f7e81ed186b2b307caca6e49e058dcb490511a038178c1011517b63
Jul 17 22:28:28.775016 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:28.774994    2565 scope.go:117] "RemoveContainer" containerID="e9863879a63d8808afea50ed09711839dbea2a5de0797ac5e2548c8a7eb0a20d"
Jul 17 22:28:28.775417 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:28:28.775253    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-skrb2_odh-ai-gateway-infra(f14bf336-6a5a-45a9-9794-77f01de59d81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2" podUID="f14bf336-6a5a-45a9-9794-77f01de59d81"
Jul 17 22:28:28.776637 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:28.776611    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh" event={"ID":"9f6a4aeb-32b1-41e3-953a-7b1241297db0","Type":"ContainerStarted","Data":"f80e843549329183ca86c25aad198d76cc4fdd5161503c0064ccde4f78b4c97c"}
Jul 17 22:28:28.777834 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:28.777812    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6" event={"ID":"73da79ed-e9dd-4af5-9819-7858900fbf5b","Type":"ContainerStarted","Data":"27c1c57f1f7e81ed186b2b307caca6e49e058dcb490511a038178c1011517b63"}
Jul 17 22:28:28.807003 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:28.806946    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh" podStartSLOduration=16.806931043 podStartE2EDuration="16.806931043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:28:28.805596628 +0000 UTC m=+2572.194613282" watchObservedRunningTime="2026-07-17 22:28:28.806931043 +0000 UTC m=+2572.195947730"
Jul 17 22:28:29.784721 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:29.784687    2565 generic.go:356] "Generic (PLEG): container finished" podID="9f6a4aeb-32b1-41e3-953a-7b1241297db0" containerID="f80e843549329183ca86c25aad198d76cc4fdd5161503c0064ccde4f78b4c97c" exitCode=6
Jul 17 22:28:29.785221 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:29.784766    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh" event={"ID":"9f6a4aeb-32b1-41e3-953a-7b1241297db0","Type":"ContainerDied","Data":"f80e843549329183ca86c25aad198d76cc4fdd5161503c0064ccde4f78b4c97c"}
Jul 17 22:28:29.785221 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:29.784849    2565 scope.go:117] "RemoveContainer" containerID="4a21e2603d8f24f28fd36d376f93c87c2a9b520905e9e0f67a768e80407dbd6d"
Jul 17 22:28:29.785221 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:29.785215    2565 scope.go:117] "RemoveContainer" containerID="f80e843549329183ca86c25aad198d76cc4fdd5161503c0064ccde4f78b4c97c"
Jul 17 22:28:29.785456 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:28:29.785434    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh_odh-ai-gateway-infra(9f6a4aeb-32b1-41e3-953a-7b1241297db0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh" podUID="9f6a4aeb-32b1-41e3-953a-7b1241297db0"
Jul 17 22:28:29.786391 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:29.786368    2565 generic.go:356] "Generic (PLEG): container finished" podID="73da79ed-e9dd-4af5-9819-7858900fbf5b" containerID="c0c7d1496b1fdaaee4be74bbbdbd752f6cdadc670e38ebce6f50663a4f167b56" exitCode=6
Jul 17 22:28:29.786493 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:29.786428    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6" event={"ID":"73da79ed-e9dd-4af5-9819-7858900fbf5b","Type":"ContainerDied","Data":"c0c7d1496b1fdaaee4be74bbbdbd752f6cdadc670e38ebce6f50663a4f167b56"}
Jul 17 22:28:29.786607 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:29.786592    2565 scope.go:117] "RemoveContainer" containerID="c0c7d1496b1fdaaee4be74bbbdbd752f6cdadc670e38ebce6f50663a4f167b56"
Jul 17 22:28:29.815310 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:29.815280    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh"]
Jul 17 22:28:29.880566 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:29.880521    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:29.954046 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:29.954011    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:30.117495 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.117458    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:30.395708 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.395668    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"]
Jul 17 22:28:30.400293 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.400267    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.403568 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.403546    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c8731528-openshift-default-dockercfg-894vx\""
Jul 17 22:28:30.411918 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.411888    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"]
Jul 17 22:28:30.435521 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.435463    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:30.518909 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.518875    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnvqt\" (UniqueName: \"kubernetes.io/projected/c5f6f404-542c-4b2e-8b13-4df557abe0bc-kube-api-access-gnvqt\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.519116 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.518915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-data\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.519116 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.518945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-credential-socket\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.519116 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.518970    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-podinfo\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.519116 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.519077    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-envoy\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.519310 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.519121    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-workload-socket\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.519310 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.519173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-token\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.519310 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.519270    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-workload-certs\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.519441 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.519320    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istiod-ca-cert\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.620407 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.620323    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-token\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.620407 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.620366    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-workload-certs\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.620407 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.620394    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istiod-ca-cert\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.620736 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.620439    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnvqt\" (UniqueName: \"kubernetes.io/projected/c5f6f404-542c-4b2e-8b13-4df557abe0bc-kube-api-access-gnvqt\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.620736 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.620466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-data\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.620736 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.620510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-credential-socket\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.620736 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.620546    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-podinfo\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.620736 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.620573    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-envoy\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.620736 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.620601    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-workload-socket\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.621017 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.620889    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-workload-certs\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.621017 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.620922    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-data\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.621017 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.620986    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-credential-socket\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.621017 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.621012    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-workload-socket\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.621323 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.621306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istiod-ca-cert\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.623157 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.623137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-envoy\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.623367 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.623349    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-podinfo\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.629504 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.629481    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnvqt\" (UniqueName: \"kubernetes.io/projected/c5f6f404-542c-4b2e-8b13-4df557abe0bc-kube-api-access-gnvqt\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.629759 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.629741    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-token\") pod \"e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") " pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.632003 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.631976    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:30.718800 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.718771    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:30.795651 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.795622    2565 generic.go:356] "Generic (PLEG): container finished" podID="73da79ed-e9dd-4af5-9819-7858900fbf5b" containerID="5f93f07897e2560372fb1395f0ac44f3e631501cfd369c8e28f72cccc54cd7bb" exitCode=6
Jul 17 22:28:30.796017 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.795682    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6" event={"ID":"73da79ed-e9dd-4af5-9819-7858900fbf5b","Type":"ContainerDied","Data":"5f93f07897e2560372fb1395f0ac44f3e631501cfd369c8e28f72cccc54cd7bb"}
Jul 17 22:28:30.796017 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.795757    2565 scope.go:117] "RemoveContainer" containerID="c0c7d1496b1fdaaee4be74bbbdbd752f6cdadc670e38ebce6f50663a4f167b56"
Jul 17 22:28:30.796017 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.795955    2565 scope.go:117] "RemoveContainer" containerID="5f93f07897e2560372fb1395f0ac44f3e631501cfd369c8e28f72cccc54cd7bb"
Jul 17 22:28:30.796259 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:28:30.796239    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6_odh-ai-gateway-infra(73da79ed-e9dd-4af5-9819-7858900fbf5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6" podUID="73da79ed-e9dd-4af5-9819-7858900fbf5b"
Jul 17 22:28:30.872029 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.872003    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"]
Jul 17 22:28:30.874360 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:28:30.874332    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f6f404_542c_4b2e_8b13_4df557abe0bc.slice/crio-5fc8afb84f7841f2e77776380c969a2137aa05a47c8c27e26eb0d0ba3ff59319 WatchSource:0}: Error finding container 5fc8afb84f7841f2e77776380c969a2137aa05a47c8c27e26eb0d0ba3ff59319: Status 404 returned error can't find the container with id 5fc8afb84f7841f2e77776380c969a2137aa05a47c8c27e26eb0d0ba3ff59319
Jul 17 22:28:30.876859 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.876815    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:28:30.876964 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.876904    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:28:30.876964 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.876944    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:28:30.925174 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:30.925145    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh"
Jul 17 22:28:31.023844 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:31.023807    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6a4aeb-32b1-41e3-953a-7b1241297db0-maas-api-service-ca\") pod \"9f6a4aeb-32b1-41e3-953a-7b1241297db0\" (UID: \"9f6a4aeb-32b1-41e3-953a-7b1241297db0\") "
Jul 17 22:28:31.024193 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:31.024170    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f6a4aeb-32b1-41e3-953a-7b1241297db0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f6a4aeb-32b1-41e3-953a-7b1241297db0" (UID: "9f6a4aeb-32b1-41e3-953a-7b1241297db0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:28:31.125495 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:31.125399    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6a4aeb-32b1-41e3-953a-7b1241297db0-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:28:31.802266 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:31.802237    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh"
Jul 17 22:28:31.802734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:31.802235    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh" event={"ID":"9f6a4aeb-32b1-41e3-953a-7b1241297db0","Type":"ContainerDied","Data":"ceca7f3e22919c5fc3afad8e8ff3549c9c7568678c1083c2e3c2844c5d25e37b"}
Jul 17 22:28:31.802734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:31.802366    2565 scope.go:117] "RemoveContainer" containerID="f80e843549329183ca86c25aad198d76cc4fdd5161503c0064ccde4f78b4c97c"
Jul 17 22:28:31.804044 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:31.804026    2565 scope.go:117] "RemoveContainer" containerID="5f93f07897e2560372fb1395f0ac44f3e631501cfd369c8e28f72cccc54cd7bb"
Jul 17 22:28:31.804384 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:28:31.804304    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6_odh-ai-gateway-infra(73da79ed-e9dd-4af5-9819-7858900fbf5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6" podUID="73da79ed-e9dd-4af5-9819-7858900fbf5b"
Jul 17 22:28:31.805930 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:31.805901    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz" event={"ID":"c5f6f404-542c-4b2e-8b13-4df557abe0bc","Type":"ContainerStarted","Data":"188c656ea7a415ecf6fbc38535acd26ff98c0bea8e99400453dfa0aeee341b91"}
Jul 17 22:28:31.806022 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:31.805947    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz" event={"ID":"c5f6f404-542c-4b2e-8b13-4df557abe0bc","Type":"ContainerStarted","Data":"5fc8afb84f7841f2e77776380c969a2137aa05a47c8c27e26eb0d0ba3ff59319"}
Jul 17 22:28:31.817045 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:31.817022    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh"]
Jul 17 22:28:31.821387 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:31.821361    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-qxwlh"]
Jul 17 22:28:31.864244 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:31.864194    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz" podStartSLOduration=1.864173331 podStartE2EDuration="1.864173331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:28:31.863300054 +0000 UTC m=+2575.252316706" watchObservedRunningTime="2026-07-17 22:28:31.864173331 +0000 UTC m=+2575.253189984"
Jul 17 22:28:32.719303 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:32.719271    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:32.724542 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:32.724516    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:32.815089 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:32.813327    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:32.819223 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:32.819193    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:33.182692 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:33.182656    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f6a4aeb-32b1-41e3-953a-7b1241297db0" path="/var/lib/kubelet/pods/9f6a4aeb-32b1-41e3-953a-7b1241297db0/volumes"
Jul 17 22:28:33.800295 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:33.800263    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:33.876466 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:33.876378    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:33.949033 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:33.948998    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:34.437375 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:34.437345    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:34.538450 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:34.538412    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:34.638640 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:34.638604    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:38.307201 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:38.307160    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:38.929897 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:38.929857    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:39.030253 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:39.030192    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:39.045175 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:39.045135    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:28:39.459298 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:39.459262    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:42.177303 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:42.177271    2565 scope.go:117] "RemoveContainer" containerID="e9863879a63d8808afea50ed09711839dbea2a5de0797ac5e2548c8a7eb0a20d"
Jul 17 22:28:42.865631 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:42.865592    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2" event={"ID":"f14bf336-6a5a-45a9-9794-77f01de59d81","Type":"ContainerStarted","Data":"d4f730753027652ecfe7aedbcf78b5cce4c2a583c6090621d2455d5ed85acb44"}
Jul 17 22:28:43.875580 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:43.875535    2565 generic.go:356] "Generic (PLEG): container finished" podID="f14bf336-6a5a-45a9-9794-77f01de59d81" containerID="d4f730753027652ecfe7aedbcf78b5cce4c2a583c6090621d2455d5ed85acb44" exitCode=6
Jul 17 22:28:43.876254 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:43.875639    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2" event={"ID":"f14bf336-6a5a-45a9-9794-77f01de59d81","Type":"ContainerDied","Data":"d4f730753027652ecfe7aedbcf78b5cce4c2a583c6090621d2455d5ed85acb44"}
Jul 17 22:28:43.876254 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:43.875731    2565 scope.go:117] "RemoveContainer" containerID="e9863879a63d8808afea50ed09711839dbea2a5de0797ac5e2548c8a7eb0a20d"
Jul 17 22:28:43.876254 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:43.876139    2565 scope.go:117] "RemoveContainer" containerID="d4f730753027652ecfe7aedbcf78b5cce4c2a583c6090621d2455d5ed85acb44"
Jul 17 22:28:43.876443 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:28:43.876390    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-skrb2_odh-ai-gateway-infra(f14bf336-6a5a-45a9-9794-77f01de59d81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2" podUID="f14bf336-6a5a-45a9-9794-77f01de59d81"
Jul 17 22:28:43.890993 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:43.890964    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2"]
Jul 17 22:28:44.176928 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:44.176844    2565 scope.go:117] "RemoveContainer" containerID="5f93f07897e2560372fb1395f0ac44f3e631501cfd369c8e28f72cccc54cd7bb"
Jul 17 22:28:44.885059 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:44.885021    2565 generic.go:356] "Generic (PLEG): container finished" podID="73da79ed-e9dd-4af5-9819-7858900fbf5b" containerID="a084b7db7941fa1859a2fd11304a3f5b9658aa68fddf219f87fcd81905153adc" exitCode=6
Jul 17 22:28:44.885513 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:44.885114    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6" event={"ID":"73da79ed-e9dd-4af5-9819-7858900fbf5b","Type":"ContainerDied","Data":"a084b7db7941fa1859a2fd11304a3f5b9658aa68fddf219f87fcd81905153adc"}
Jul 17 22:28:44.885513 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:44.885196    2565 scope.go:117] "RemoveContainer" containerID="5f93f07897e2560372fb1395f0ac44f3e631501cfd369c8e28f72cccc54cd7bb"
Jul 17 22:28:44.885646 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:44.885553    2565 scope.go:117] "RemoveContainer" containerID="a084b7db7941fa1859a2fd11304a3f5b9658aa68fddf219f87fcd81905153adc"
Jul 17 22:28:44.885888 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:28:44.885816    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6_odh-ai-gateway-infra(73da79ed-e9dd-4af5-9819-7858900fbf5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6" podUID="73da79ed-e9dd-4af5-9819-7858900fbf5b"
Jul 17 22:28:45.040593 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.040565    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2"
Jul 17 22:28:45.048735 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.048703    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f14bf336-6a5a-45a9-9794-77f01de59d81-maas-api-service-ca\") pod \"f14bf336-6a5a-45a9-9794-77f01de59d81\" (UID: \"f14bf336-6a5a-45a9-9794-77f01de59d81\") "
Jul 17 22:28:45.049172 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.049147    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14bf336-6a5a-45a9-9794-77f01de59d81-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f14bf336-6a5a-45a9-9794-77f01de59d81" (UID: "f14bf336-6a5a-45a9-9794-77f01de59d81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:28:45.150397 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.150307    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f14bf336-6a5a-45a9-9794-77f01de59d81-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:28:45.256087 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.256017    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:45.325286 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.325245    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:45.621735 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.621698    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c8731528-ztj4x"]
Jul 17 22:28:45.622084 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.622059    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f14bf336-6a5a-45a9-9794-77f01de59d81" containerName="revoke-keys"
Jul 17 22:28:45.622156 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.622085    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f14bf336-6a5a-45a9-9794-77f01de59d81" containerName="revoke-keys"
Jul 17 22:28:45.622156 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.622097    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f6a4aeb-32b1-41e3-953a-7b1241297db0" containerName="revoke-keys"
Jul 17 22:28:45.622156 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.622104    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f6a4aeb-32b1-41e3-953a-7b1241297db0" containerName="revoke-keys"
Jul 17 22:28:45.622156 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.622110    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f6a4aeb-32b1-41e3-953a-7b1241297db0" containerName="revoke-keys"
Jul 17 22:28:45.622156 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.622115    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f6a4aeb-32b1-41e3-953a-7b1241297db0" containerName="revoke-keys"
Jul 17 22:28:45.622156 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.622135    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f14bf336-6a5a-45a9-9794-77f01de59d81" containerName="revoke-keys"
Jul 17 22:28:45.622156 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.622148    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f14bf336-6a5a-45a9-9794-77f01de59d81" containerName="revoke-keys"
Jul 17 22:28:45.622406 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.622201    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f6a4aeb-32b1-41e3-953a-7b1241297db0" containerName="revoke-keys"
Jul 17 22:28:45.622406 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.622210    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f14bf336-6a5a-45a9-9794-77f01de59d81" containerName="revoke-keys"
Jul 17 22:28:45.622406 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.622215    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f6a4aeb-32b1-41e3-953a-7b1241297db0" containerName="revoke-keys"
Jul 17 22:28:45.622406 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.622222    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f14bf336-6a5a-45a9-9794-77f01de59d81" containerName="revoke-keys"
Jul 17 22:28:45.622406 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.622229    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f14bf336-6a5a-45a9-9794-77f01de59d81" containerName="revoke-keys"
Jul 17 22:28:45.626127 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.626105    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c8731528-ztj4x"
Jul 17 22:28:45.633411 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.633375    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c8731528-ztj4x"]
Jul 17 22:28:45.655541 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.655500    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d80a957c-30ac-44e2-a1dc-f8d233b8a189-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c8731528-ztj4x\" (UID: \"d80a957c-30ac-44e2-a1dc-f8d233b8a189\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c8731528-ztj4x"
Jul 17 22:28:45.756370 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.756328    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d80a957c-30ac-44e2-a1dc-f8d233b8a189-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c8731528-ztj4x\" (UID: \"d80a957c-30ac-44e2-a1dc-f8d233b8a189\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c8731528-ztj4x"
Jul 17 22:28:45.756938 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.756918    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d80a957c-30ac-44e2-a1dc-f8d233b8a189-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c8731528-ztj4x\" (UID: \"d80a957c-30ac-44e2-a1dc-f8d233b8a189\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c8731528-ztj4x"
Jul 17 22:28:45.891761 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.891668    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2"
Jul 17 22:28:45.891761 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.891697    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2" event={"ID":"f14bf336-6a5a-45a9-9794-77f01de59d81","Type":"ContainerDied","Data":"3bc2b46882daa1682dfbade0e23838b05fb3aef1696614ea12ddda71bc131a3b"}
Jul 17 22:28:45.892339 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.891764    2565 scope.go:117] "RemoveContainer" containerID="d4f730753027652ecfe7aedbcf78b5cce4c2a583c6090621d2455d5ed85acb44"
Jul 17 22:28:45.912485 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.912450    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2"]
Jul 17 22:28:45.917982 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.917952    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-skrb2"]
Jul 17 22:28:45.922894 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.922866    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6"]
Jul 17 22:28:45.938098 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:45.938054    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c8731528-ztj4x"
Jul 17 22:28:46.114590 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.114560    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6"
Jul 17 22:28:46.137350 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.136753    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c8731528-ztj4x"]
Jul 17 22:28:46.139874 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:28:46.139841    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80a957c_30ac_44e2_a1dc_f8d233b8a189.slice/crio-634a633e7d6895d40832d3b4d9df29d257f2fd2c6fc22367e8b4f99c630339e4 WatchSource:0}: Error finding container 634a633e7d6895d40832d3b4d9df29d257f2fd2c6fc22367e8b4f99c630339e4: Status 404 returned error can't find the container with id 634a633e7d6895d40832d3b4d9df29d257f2fd2c6fc22367e8b4f99c630339e4
Jul 17 22:28:46.159898 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.159875    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73da79ed-e9dd-4af5-9819-7858900fbf5b-maas-api-service-ca\") pod \"73da79ed-e9dd-4af5-9819-7858900fbf5b\" (UID: \"73da79ed-e9dd-4af5-9819-7858900fbf5b\") "
Jul 17 22:28:46.160339 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.160313    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73da79ed-e9dd-4af5-9819-7858900fbf5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73da79ed-e9dd-4af5-9819-7858900fbf5b" (UID: "73da79ed-e9dd-4af5-9819-7858900fbf5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:28:46.261562 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.261531    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73da79ed-e9dd-4af5-9819-7858900fbf5b-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:28:46.900005 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.899968    2565 generic.go:356] "Generic (PLEG): container finished" podID="d80a957c-30ac-44e2-a1dc-f8d233b8a189" containerID="108ccfde6d693a886e219edbb576972e613509b059ef837f9c1d19df17c853c0" exitCode=0
Jul 17 22:28:46.900547 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.900055    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c8731528-ztj4x" event={"ID":"d80a957c-30ac-44e2-a1dc-f8d233b8a189","Type":"ContainerDied","Data":"108ccfde6d693a886e219edbb576972e613509b059ef837f9c1d19df17c853c0"}
Jul 17 22:28:46.900547 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.900135    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c8731528-ztj4x" event={"ID":"d80a957c-30ac-44e2-a1dc-f8d233b8a189","Type":"ContainerStarted","Data":"634a633e7d6895d40832d3b4d9df29d257f2fd2c6fc22367e8b4f99c630339e4"}
Jul 17 22:28:46.902221 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.902203    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6"
Jul 17 22:28:46.902221 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.902212    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6" event={"ID":"73da79ed-e9dd-4af5-9819-7858900fbf5b","Type":"ContainerDied","Data":"27c1c57f1f7e81ed186b2b307caca6e49e058dcb490511a038178c1011517b63"}
Jul 17 22:28:46.902383 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.902272    2565 scope.go:117] "RemoveContainer" containerID="a084b7db7941fa1859a2fd11304a3f5b9658aa68fddf219f87fcd81905153adc"
Jul 17 22:28:46.904558 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.904534    2565 generic.go:356] "Generic (PLEG): container finished" podID="c5f6f404-542c-4b2e-8b13-4df557abe0bc" containerID="188c656ea7a415ecf6fbc38535acd26ff98c0bea8e99400453dfa0aeee341b91" exitCode=0
Jul 17 22:28:46.904668 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.904611    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz" event={"ID":"c5f6f404-542c-4b2e-8b13-4df557abe0bc","Type":"ContainerDied","Data":"188c656ea7a415ecf6fbc38535acd26ff98c0bea8e99400453dfa0aeee341b91"}
Jul 17 22:28:46.905024 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.905008    2565 scope.go:117] "RemoveContainer" containerID="188c656ea7a415ecf6fbc38535acd26ff98c0bea8e99400453dfa0aeee341b91"
Jul 17 22:28:46.906177 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.906138    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:28:46.906285 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.906236    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:28:46.906355 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.906291    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:28:46.928433 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.928407    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6"]
Jul 17 22:28:46.936164 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:46.936136    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-nfpf6"]
Jul 17 22:28:47.190306 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:47.186045    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73da79ed-e9dd-4af5-9819-7858900fbf5b" path="/var/lib/kubelet/pods/73da79ed-e9dd-4af5-9819-7858900fbf5b/volumes"
Jul 17 22:28:47.190306 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:47.186612    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f14bf336-6a5a-45a9-9794-77f01de59d81" path="/var/lib/kubelet/pods/f14bf336-6a5a-45a9-9794-77f01de59d81/volumes"
Jul 17 22:28:47.813353 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:47.813315    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:47.912147 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:47.912105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz" event={"ID":"c5f6f404-542c-4b2e-8b13-4df557abe0bc","Type":"ContainerStarted","Data":"d2e933d6272cf3f56c16b43e27849a7dd0a92504c094957fc95b5046a0e6a765"}
Jul 17 22:28:48.096834 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:48.096804    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c8731528-ztj4x"
Jul 17 22:28:48.182673 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:48.182625    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d80a957c-30ac-44e2-a1dc-f8d233b8a189-maas-api-service-ca\") pod \"d80a957c-30ac-44e2-a1dc-f8d233b8a189\" (UID: \"d80a957c-30ac-44e2-a1dc-f8d233b8a189\") "
Jul 17 22:28:48.182988 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:48.182963    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d80a957c-30ac-44e2-a1dc-f8d233b8a189-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d80a957c-30ac-44e2-a1dc-f8d233b8a189" (UID: "d80a957c-30ac-44e2-a1dc-f8d233b8a189"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:28:48.283393 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:48.283345    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d80a957c-30ac-44e2-a1dc-f8d233b8a189-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:28:48.719903 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:48.719861    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:48.720953 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:48.720931    2565 patch_prober.go:28] interesting pod/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Jul 17 22:28:48.721010 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:48.720980    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz" podUID="c5f6f404-542c-4b2e-8b13-4df557abe0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Jul 17 22:28:48.918653 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:48.918622    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c8731528-ztj4x"
Jul 17 22:28:48.919184 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:48.918657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c8731528-ztj4x" event={"ID":"d80a957c-30ac-44e2-a1dc-f8d233b8a189","Type":"ContainerDied","Data":"634a633e7d6895d40832d3b4d9df29d257f2fd2c6fc22367e8b4f99c630339e4"}
Jul 17 22:28:48.919184 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:48.918708    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="634a633e7d6895d40832d3b4d9df29d257f2fd2c6fc22367e8b4f99c630339e4"
Jul 17 22:28:49.720044 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:49.720004    2565 patch_prober.go:28] interesting pod/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Jul 17 22:28:49.720281 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:49.720089    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz" podUID="c5f6f404-542c-4b2e-8b13-4df557abe0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Jul 17 22:28:50.719997 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:50.719957    2565 patch_prober.go:28] interesting pod/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Jul 17 22:28:50.720419 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:50.720037    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz" podUID="c5f6f404-542c-4b2e-8b13-4df557abe0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Jul 17 22:28:51.720144 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:51.720103    2565 patch_prober.go:28] interesting pod/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Jul 17 22:28:51.720583 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:51.720164    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz" podUID="c5f6f404-542c-4b2e-8b13-4df557abe0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Jul 17 22:28:52.723896 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:52.723861    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:52.724443 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:52.724196    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:52.725043 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:52.725026    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:28:56.040263 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.040230    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:28:56.277303 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.277262    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8"]
Jul 17 22:28:56.277715 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.277696    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73da79ed-e9dd-4af5-9819-7858900fbf5b" containerName="revoke-keys"
Jul 17 22:28:56.277809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.277718    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73da79ed-e9dd-4af5-9819-7858900fbf5b" containerName="revoke-keys"
Jul 17 22:28:56.277809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.277748    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f14bf336-6a5a-45a9-9794-77f01de59d81" containerName="revoke-keys"
Jul 17 22:28:56.277809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.277756    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f14bf336-6a5a-45a9-9794-77f01de59d81" containerName="revoke-keys"
Jul 17 22:28:56.277809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.277773    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f6a4aeb-32b1-41e3-953a-7b1241297db0" containerName="revoke-keys"
Jul 17 22:28:56.277809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.277783    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f6a4aeb-32b1-41e3-953a-7b1241297db0" containerName="revoke-keys"
Jul 17 22:28:56.277809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.277794    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d80a957c-30ac-44e2-a1dc-f8d233b8a189" containerName="revoke-keys"
Jul 17 22:28:56.277809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.277803    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d80a957c-30ac-44e2-a1dc-f8d233b8a189" containerName="revoke-keys"
Jul 17 22:28:56.278209 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.277819    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73da79ed-e9dd-4af5-9819-7858900fbf5b" containerName="revoke-keys"
Jul 17 22:28:56.278209 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.277827    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73da79ed-e9dd-4af5-9819-7858900fbf5b" containerName="revoke-keys"
Jul 17 22:28:56.278209 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.277909    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="73da79ed-e9dd-4af5-9819-7858900fbf5b" containerName="revoke-keys"
Jul 17 22:28:56.278209 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.277920    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="73da79ed-e9dd-4af5-9819-7858900fbf5b" containerName="revoke-keys"
Jul 17 22:28:56.278209 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.277929    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f6a4aeb-32b1-41e3-953a-7b1241297db0" containerName="revoke-keys"
Jul 17 22:28:56.278209 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.277942    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d80a957c-30ac-44e2-a1dc-f8d233b8a189" containerName="revoke-keys"
Jul 17 22:28:56.281592 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.281572    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8"
Jul 17 22:28:56.286207 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.286183    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:28:56.286339 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.286203    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:28:56.292676 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.292606    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8"]
Jul 17 22:28:56.340507 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.340478    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7fdce7-fece-47ed-ae95-fd400878a136-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8\" (UID: \"da7fdce7-fece-47ed-ae95-fd400878a136\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8"
Jul 17 22:28:56.441347 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.441303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7fdce7-fece-47ed-ae95-fd400878a136-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8\" (UID: \"da7fdce7-fece-47ed-ae95-fd400878a136\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8"
Jul 17 22:28:56.441983 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.441955    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7fdce7-fece-47ed-ae95-fd400878a136-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8\" (UID: \"da7fdce7-fece-47ed-ae95-fd400878a136\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8"
Jul 17 22:28:56.593967 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.593867    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8"
Jul 17 22:28:56.732748 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.732721    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8"]
Jul 17 22:28:56.735389 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:28:56.735353    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7fdce7_fece_47ed_ae95_fd400878a136.slice/crio-fd117c035dc4e2c57fa2035a36a447ec718968e8166171bbe701a9f7be9f3c2e WatchSource:0}: Error finding container fd117c035dc4e2c57fa2035a36a447ec718968e8166171bbe701a9f7be9f3c2e: Status 404 returned error can't find the container with id fd117c035dc4e2c57fa2035a36a447ec718968e8166171bbe701a9f7be9f3c2e
Jul 17 22:28:56.957793 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:56.957703    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8" event={"ID":"da7fdce7-fece-47ed-ae95-fd400878a136","Type":"ContainerStarted","Data":"fd117c035dc4e2c57fa2035a36a447ec718968e8166171bbe701a9f7be9f3c2e"}
Jul 17 22:28:57.963746 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:57.963651    2565 generic.go:356] "Generic (PLEG): container finished" podID="da7fdce7-fece-47ed-ae95-fd400878a136" containerID="cf4e4e3ef13adc8ff6498caf22bed550ba5c3a7e7cc9a2e883d7d3d07aea875c" exitCode=6
Jul 17 22:28:57.963746 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:57.963724    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8" event={"ID":"da7fdce7-fece-47ed-ae95-fd400878a136","Type":"ContainerDied","Data":"cf4e4e3ef13adc8ff6498caf22bed550ba5c3a7e7cc9a2e883d7d3d07aea875c"}
Jul 17 22:28:57.964192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:57.963894    2565 scope.go:117] "RemoveContainer" containerID="cf4e4e3ef13adc8ff6498caf22bed550ba5c3a7e7cc9a2e883d7d3d07aea875c"
Jul 17 22:28:58.970979 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:58.970889    2565 generic.go:356] "Generic (PLEG): container finished" podID="da7fdce7-fece-47ed-ae95-fd400878a136" containerID="5d60f7fb9a46194c9ac110f42390d045ad39c05bac531489505dff34e56fc1cb" exitCode=6
Jul 17 22:28:58.971443 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:58.971030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8" event={"ID":"da7fdce7-fece-47ed-ae95-fd400878a136","Type":"ContainerDied","Data":"5d60f7fb9a46194c9ac110f42390d045ad39c05bac531489505dff34e56fc1cb"}
Jul 17 22:28:58.971443 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:58.971120    2565 scope.go:117] "RemoveContainer" containerID="cf4e4e3ef13adc8ff6498caf22bed550ba5c3a7e7cc9a2e883d7d3d07aea875c"
Jul 17 22:28:58.971443 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:58.971339    2565 scope.go:117] "RemoveContainer" containerID="5d60f7fb9a46194c9ac110f42390d045ad39c05bac531489505dff34e56fc1cb"
Jul 17 22:28:58.971573 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:28:58.971555    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8_odh-ai-gateway-infra(da7fdce7-fece-47ed-ae95-fd400878a136)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8" podUID="da7fdce7-fece-47ed-ae95-fd400878a136"
Jul 17 22:28:59.037940 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:59.037900    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:28:59.977963 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:28:59.977936    2565 scope.go:117] "RemoveContainer" containerID="5d60f7fb9a46194c9ac110f42390d045ad39c05bac531489505dff34e56fc1cb"
Jul 17 22:28:59.978421 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:28:59.978170    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8_odh-ai-gateway-infra(da7fdce7-fece-47ed-ae95-fd400878a136)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8" podUID="da7fdce7-fece-47ed-ae95-fd400878a136"
Jul 17 22:29:00.702502 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:00.702452    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:02.630470 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:02.630434    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c"]
Jul 17 22:29:02.630870 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:02.630804    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73da79ed-e9dd-4af5-9819-7858900fbf5b" containerName="revoke-keys"
Jul 17 22:29:02.630870 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:02.630815    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73da79ed-e9dd-4af5-9819-7858900fbf5b" containerName="revoke-keys"
Jul 17 22:29:02.630944 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:02.630886    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="73da79ed-e9dd-4af5-9819-7858900fbf5b" containerName="revoke-keys"
Jul 17 22:29:02.635283 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:02.635262    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c"
Jul 17 22:29:02.641237 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:02.641202    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c"]
Jul 17 22:29:02.697882 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:02.697839    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381b35f1-f23d-4fad-8887-875cedd82e76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c\" (UID: \"381b35f1-f23d-4fad-8887-875cedd82e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c"
Jul 17 22:29:02.798879 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:02.798839    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381b35f1-f23d-4fad-8887-875cedd82e76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c\" (UID: \"381b35f1-f23d-4fad-8887-875cedd82e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c"
Jul 17 22:29:02.799511 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:02.799490    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381b35f1-f23d-4fad-8887-875cedd82e76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c\" (UID: \"381b35f1-f23d-4fad-8887-875cedd82e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c"
Jul 17 22:29:02.947856 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:02.947766    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c"
Jul 17 22:29:03.088969 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:03.088941    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c"]
Jul 17 22:29:03.090814 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:29:03.090787    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381b35f1_f23d_4fad_8887_875cedd82e76.slice/crio-94f023d7bbc80cdb822e996f4ff6f3a41d9f83bc01747269b88455a7cf59adb8 WatchSource:0}: Error finding container 94f023d7bbc80cdb822e996f4ff6f3a41d9f83bc01747269b88455a7cf59adb8: Status 404 returned error can't find the container with id 94f023d7bbc80cdb822e996f4ff6f3a41d9f83bc01747269b88455a7cf59adb8
Jul 17 22:29:04.003636 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:04.003606    2565 generic.go:356] "Generic (PLEG): container finished" podID="381b35f1-f23d-4fad-8887-875cedd82e76" containerID="972ace472d8a98f82a8f2f82d7f4294b7aed98ec22c65b00a94b855f502c9caa" exitCode=6
Jul 17 22:29:04.004013 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:04.003643    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c" event={"ID":"381b35f1-f23d-4fad-8887-875cedd82e76","Type":"ContainerDied","Data":"972ace472d8a98f82a8f2f82d7f4294b7aed98ec22c65b00a94b855f502c9caa"}
Jul 17 22:29:04.004013 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:04.003680    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c" event={"ID":"381b35f1-f23d-4fad-8887-875cedd82e76","Type":"ContainerStarted","Data":"94f023d7bbc80cdb822e996f4ff6f3a41d9f83bc01747269b88455a7cf59adb8"}
Jul 17 22:29:04.004013 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:04.003875    2565 scope.go:117] "RemoveContainer" containerID="972ace472d8a98f82a8f2f82d7f4294b7aed98ec22c65b00a94b855f502c9caa"
Jul 17 22:29:04.847142 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:04.847037    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:04.947960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:04.947926    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:05.010273 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:05.010234    2565 generic.go:356] "Generic (PLEG): container finished" podID="381b35f1-f23d-4fad-8887-875cedd82e76" containerID="9e55eb385b534fa0d37b8ef023f963dbd7541b8627a6fa4be457618f5f9b12c0" exitCode=6
Jul 17 22:29:05.010686 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:05.010332    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c" event={"ID":"381b35f1-f23d-4fad-8887-875cedd82e76","Type":"ContainerDied","Data":"9e55eb385b534fa0d37b8ef023f963dbd7541b8627a6fa4be457618f5f9b12c0"}
Jul 17 22:29:05.010686 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:05.010399    2565 scope.go:117] "RemoveContainer" containerID="972ace472d8a98f82a8f2f82d7f4294b7aed98ec22c65b00a94b855f502c9caa"
Jul 17 22:29:05.010686 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:05.010590    2565 scope.go:117] "RemoveContainer" containerID="9e55eb385b534fa0d37b8ef023f963dbd7541b8627a6fa4be457618f5f9b12c0"
Jul 17 22:29:05.010882 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:29:05.010858    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c_odh-ai-gateway-infra(381b35f1-f23d-4fad-8887-875cedd82e76)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c" podUID="381b35f1-f23d-4fad-8887-875cedd82e76"
Jul 17 22:29:06.016979 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:06.016951    2565 scope.go:117] "RemoveContainer" containerID="9e55eb385b534fa0d37b8ef023f963dbd7541b8627a6fa4be457618f5f9b12c0"
Jul 17 22:29:06.017379 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:29:06.017200    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c_odh-ai-gateway-infra(381b35f1-f23d-4fad-8887-875cedd82e76)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c" podUID="381b35f1-f23d-4fad-8887-875cedd82e76"
Jul 17 22:29:06.480901 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:06.480806    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:06.542040 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:06.541999    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:10.176650 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:10.176620    2565 scope.go:117] "RemoveContainer" containerID="5d60f7fb9a46194c9ac110f42390d045ad39c05bac531489505dff34e56fc1cb"
Jul 17 22:29:11.043027 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:11.042947    2565 generic.go:356] "Generic (PLEG): container finished" podID="da7fdce7-fece-47ed-ae95-fd400878a136" containerID="9d90cc62ac71d494dde117744eebaf8f65a747aa9cc7da6a48396e25934af8f5" exitCode=6
Jul 17 22:29:11.043196 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:11.043022    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8" event={"ID":"da7fdce7-fece-47ed-ae95-fd400878a136","Type":"ContainerDied","Data":"9d90cc62ac71d494dde117744eebaf8f65a747aa9cc7da6a48396e25934af8f5"}
Jul 17 22:29:11.043196 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:11.043105    2565 scope.go:117] "RemoveContainer" containerID="5d60f7fb9a46194c9ac110f42390d045ad39c05bac531489505dff34e56fc1cb"
Jul 17 22:29:11.043478 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:11.043463    2565 scope.go:117] "RemoveContainer" containerID="9d90cc62ac71d494dde117744eebaf8f65a747aa9cc7da6a48396e25934af8f5"
Jul 17 22:29:11.043692 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:29:11.043671    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8_odh-ai-gateway-infra(da7fdce7-fece-47ed-ae95-fd400878a136)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8" podUID="da7fdce7-fece-47ed-ae95-fd400878a136"
Jul 17 22:29:11.426832 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:11.426732    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"]
Jul 17 22:29:11.427322 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:11.426955    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz" podUID="c5f6f404-542c-4b2e-8b13-4df557abe0bc" containerName="istio-proxy" containerID="cri-o://d2e933d6272cf3f56c16b43e27849a7dd0a92504c094957fc95b5046a0e6a765" gracePeriod=30
Jul 17 22:29:11.451642 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:11.451612    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:11.688134 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:11.688019    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:11.813985 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:11.813946    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:11.913696 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:11.913649    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:12.039258 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:12.039221    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:12.074308 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:12.074274    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8"]
Jul 17 22:29:12.205365 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:12.205330    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8"
Jul 17 22:29:12.287834 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:12.287790    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7fdce7-fece-47ed-ae95-fd400878a136-maas-api-service-ca\") pod \"da7fdce7-fece-47ed-ae95-fd400878a136\" (UID: \"da7fdce7-fece-47ed-ae95-fd400878a136\") "
Jul 17 22:29:12.288157 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:12.288130    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7fdce7-fece-47ed-ae95-fd400878a136-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da7fdce7-fece-47ed-ae95-fd400878a136" (UID: "da7fdce7-fece-47ed-ae95-fd400878a136"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:29:12.389528 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:12.389438    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7fdce7-fece-47ed-ae95-fd400878a136-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:29:13.055407 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:13.055369    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8" event={"ID":"da7fdce7-fece-47ed-ae95-fd400878a136","Type":"ContainerDied","Data":"fd117c035dc4e2c57fa2035a36a447ec718968e8166171bbe701a9f7be9f3c2e"}
Jul 17 22:29:13.055407 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:13.055405    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8"
Jul 17 22:29:13.055925 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:13.055438    2565 scope.go:117] "RemoveContainer" containerID="9d90cc62ac71d494dde117744eebaf8f65a747aa9cc7da6a48396e25934af8f5"
Jul 17 22:29:13.079253 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:13.079220    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8"]
Jul 17 22:29:13.085246 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:13.085215    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wgvc8"]
Jul 17 22:29:13.184675 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:13.184644    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da7fdce7-fece-47ed-ae95-fd400878a136" path="/var/lib/kubelet/pods/da7fdce7-fece-47ed-ae95-fd400878a136/volumes"
Jul 17 22:29:16.694696 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.694634    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:29:16.828481 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.828441    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-data\") pod \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") "
Jul 17 22:29:16.828688 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.828483    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-envoy\") pod \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") "
Jul 17 22:29:16.828688 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.828522    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-podinfo\") pod \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") "
Jul 17 22:29:16.828688 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.828542    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-workload-certs\") pod \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") "
Jul 17 22:29:16.828688 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.828579    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-credential-socket\") pod \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") "
Jul 17 22:29:16.828688 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.828622    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-token\") pod \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") "
Jul 17 22:29:16.828688 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.828680    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnvqt\" (UniqueName: \"kubernetes.io/projected/c5f6f404-542c-4b2e-8b13-4df557abe0bc-kube-api-access-gnvqt\") pod \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") "
Jul 17 22:29:16.829018 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.828707    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-workload-socket\") pod \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") "
Jul 17 22:29:16.829018 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.828781    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istiod-ca-cert\") pod \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\" (UID: \"c5f6f404-542c-4b2e-8b13-4df557abe0bc\") "
Jul 17 22:29:16.829018 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.828893    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c5f6f404-542c-4b2e-8b13-4df557abe0bc" (UID: "c5f6f404-542c-4b2e-8b13-4df557abe0bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:29:16.829018 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.828908    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c5f6f404-542c-4b2e-8b13-4df557abe0bc" (UID: "c5f6f404-542c-4b2e-8b13-4df557abe0bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:29:16.829281 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.829136    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-credential-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:29:16.829281 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.829152    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c5f6f404-542c-4b2e-8b13-4df557abe0bc" (UID: "c5f6f404-542c-4b2e-8b13-4df557abe0bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:29:16.829281 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.829157    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-workload-certs\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:29:16.829281 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.829226    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c5f6f404-542c-4b2e-8b13-4df557abe0bc" (UID: "c5f6f404-542c-4b2e-8b13-4df557abe0bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:29:16.829594 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.829314    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "c5f6f404-542c-4b2e-8b13-4df557abe0bc" (UID: "c5f6f404-542c-4b2e-8b13-4df557abe0bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:29:16.830998 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.830971    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c5f6f404-542c-4b2e-8b13-4df557abe0bc" (UID: "c5f6f404-542c-4b2e-8b13-4df557abe0bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:29:16.831197 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.831049    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c5f6f404-542c-4b2e-8b13-4df557abe0bc" (UID: "c5f6f404-542c-4b2e-8b13-4df557abe0bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:29:16.831502 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.831480    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "c5f6f404-542c-4b2e-8b13-4df557abe0bc" (UID: "c5f6f404-542c-4b2e-8b13-4df557abe0bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:29:16.831550 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.831506    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f6f404-542c-4b2e-8b13-4df557abe0bc-kube-api-access-gnvqt" (OuterVolumeSpecName: "kube-api-access-gnvqt") pod "c5f6f404-542c-4b2e-8b13-4df557abe0bc" (UID: "c5f6f404-542c-4b2e-8b13-4df557abe0bc"). InnerVolumeSpecName "kube-api-access-gnvqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:29:16.929850 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.929810    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istiod-ca-cert\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:29:16.929850 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.929842    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-data\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:29:16.929850 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.929851    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-envoy\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:29:16.929850 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.929858    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-podinfo\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:29:16.929850 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.929866    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f6f404-542c-4b2e-8b13-4df557abe0bc-istio-token\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:29:16.930206 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.929874    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnvqt\" (UniqueName: \"kubernetes.io/projected/c5f6f404-542c-4b2e-8b13-4df557abe0bc-kube-api-access-gnvqt\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:29:16.930206 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:16.929884    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f6f404-542c-4b2e-8b13-4df557abe0bc-workload-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:29:17.076996 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:17.076960    2565 generic.go:356] "Generic (PLEG): container finished" podID="c5f6f404-542c-4b2e-8b13-4df557abe0bc" containerID="d2e933d6272cf3f56c16b43e27849a7dd0a92504c094957fc95b5046a0e6a765" exitCode=0
Jul 17 22:29:17.077191 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:17.077025    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"
Jul 17 22:29:17.077191 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:17.077043    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz" event={"ID":"c5f6f404-542c-4b2e-8b13-4df557abe0bc","Type":"ContainerDied","Data":"d2e933d6272cf3f56c16b43e27849a7dd0a92504c094957fc95b5046a0e6a765"}
Jul 17 22:29:17.077191 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:17.077148    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz" event={"ID":"c5f6f404-542c-4b2e-8b13-4df557abe0bc","Type":"ContainerDied","Data":"5fc8afb84f7841f2e77776380c969a2137aa05a47c8c27e26eb0d0ba3ff59319"}
Jul 17 22:29:17.077191 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:17.077171    2565 scope.go:117] "RemoveContainer" containerID="d2e933d6272cf3f56c16b43e27849a7dd0a92504c094957fc95b5046a0e6a765"
Jul 17 22:29:17.086589 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:17.086570    2565 scope.go:117] "RemoveContainer" containerID="188c656ea7a415ecf6fbc38535acd26ff98c0bea8e99400453dfa0aeee341b91"
Jul 17 22:29:17.094768 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:17.094747    2565 scope.go:117] "RemoveContainer" containerID="d2e933d6272cf3f56c16b43e27849a7dd0a92504c094957fc95b5046a0e6a765"
Jul 17 22:29:17.095031 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:29:17.095009    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2e933d6272cf3f56c16b43e27849a7dd0a92504c094957fc95b5046a0e6a765\": container with ID starting with d2e933d6272cf3f56c16b43e27849a7dd0a92504c094957fc95b5046a0e6a765 not found: ID does not exist" containerID="d2e933d6272cf3f56c16b43e27849a7dd0a92504c094957fc95b5046a0e6a765"
Jul 17 22:29:17.095210 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:17.095044    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2e933d6272cf3f56c16b43e27849a7dd0a92504c094957fc95b5046a0e6a765"} err="failed to get container status \"d2e933d6272cf3f56c16b43e27849a7dd0a92504c094957fc95b5046a0e6a765\": rpc error: code = NotFound desc = could not find container \"d2e933d6272cf3f56c16b43e27849a7dd0a92504c094957fc95b5046a0e6a765\": container with ID starting with d2e933d6272cf3f56c16b43e27849a7dd0a92504c094957fc95b5046a0e6a765 not found: ID does not exist"
Jul 17 22:29:17.095210 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:17.095098    2565 scope.go:117] "RemoveContainer" containerID="188c656ea7a415ecf6fbc38535acd26ff98c0bea8e99400453dfa0aeee341b91"
Jul 17 22:29:17.095385 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:29:17.095368    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188c656ea7a415ecf6fbc38535acd26ff98c0bea8e99400453dfa0aeee341b91\": container with ID starting with 188c656ea7a415ecf6fbc38535acd26ff98c0bea8e99400453dfa0aeee341b91 not found: ID does not exist" containerID="188c656ea7a415ecf6fbc38535acd26ff98c0bea8e99400453dfa0aeee341b91"
Jul 17 22:29:17.095424 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:17.095391    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188c656ea7a415ecf6fbc38535acd26ff98c0bea8e99400453dfa0aeee341b91"} err="failed to get container status \"188c656ea7a415ecf6fbc38535acd26ff98c0bea8e99400453dfa0aeee341b91\": rpc error: code = NotFound desc = could not find container \"188c656ea7a415ecf6fbc38535acd26ff98c0bea8e99400453dfa0aeee341b91\": container with ID starting with 188c656ea7a415ecf6fbc38535acd26ff98c0bea8e99400453dfa0aeee341b91 not found: ID does not exist"
Jul 17 22:29:17.100255 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:17.100230    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"]
Jul 17 22:29:17.105949 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:17.105921    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c8731528-openshift-default-7ccb44ff95-7wxxz"]
Jul 17 22:29:17.182270 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:17.182240    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5f6f404-542c-4b2e-8b13-4df557abe0bc" path="/var/lib/kubelet/pods/c5f6f404-542c-4b2e-8b13-4df557abe0bc/volumes"
Jul 17 22:29:17.939888 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:17.939855    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:18.049755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.049722    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:18.074316 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.074283    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl"]
Jul 17 22:29:18.074697 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.074684    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5f6f404-542c-4b2e-8b13-4df557abe0bc" containerName="istio-proxy"
Jul 17 22:29:18.074747 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.074699    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5f6f404-542c-4b2e-8b13-4df557abe0bc" containerName="istio-proxy"
Jul 17 22:29:18.074747 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.074709    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da7fdce7-fece-47ed-ae95-fd400878a136" containerName="revoke-keys"
Jul 17 22:29:18.074747 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.074718    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da7fdce7-fece-47ed-ae95-fd400878a136" containerName="revoke-keys"
Jul 17 22:29:18.074747 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.074729    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da7fdce7-fece-47ed-ae95-fd400878a136" containerName="revoke-keys"
Jul 17 22:29:18.074747 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.074734    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da7fdce7-fece-47ed-ae95-fd400878a136" containerName="revoke-keys"
Jul 17 22:29:18.074747 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.074743    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5f6f404-542c-4b2e-8b13-4df557abe0bc" containerName="istio-proxy"
Jul 17 22:29:18.074747 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.074748    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5f6f404-542c-4b2e-8b13-4df557abe0bc" containerName="istio-proxy"
Jul 17 22:29:18.074960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.074757    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da7fdce7-fece-47ed-ae95-fd400878a136" containerName="revoke-keys"
Jul 17 22:29:18.074960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.074762    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da7fdce7-fece-47ed-ae95-fd400878a136" containerName="revoke-keys"
Jul 17 22:29:18.074960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.074822    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da7fdce7-fece-47ed-ae95-fd400878a136" containerName="revoke-keys"
Jul 17 22:29:18.074960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.074830    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da7fdce7-fece-47ed-ae95-fd400878a136" containerName="revoke-keys"
Jul 17 22:29:18.074960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.074837    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5f6f404-542c-4b2e-8b13-4df557abe0bc" containerName="istio-proxy"
Jul 17 22:29:18.074960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.074843    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da7fdce7-fece-47ed-ae95-fd400878a136" containerName="revoke-keys"
Jul 17 22:29:18.079375 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.079352    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl"
Jul 17 22:29:18.104816 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.104789    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl"]
Jul 17 22:29:18.140691 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.140654    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5edf49-030a-417f-9f33-ed2b091ece85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl\" (UID: \"fb5edf49-030a-417f-9f33-ed2b091ece85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl"
Jul 17 22:29:18.156560 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.156526    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:18.176961 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.176935    2565 scope.go:117] "RemoveContainer" containerID="9e55eb385b534fa0d37b8ef023f963dbd7541b8627a6fa4be457618f5f9b12c0"
Jul 17 22:29:18.241233 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.241198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5edf49-030a-417f-9f33-ed2b091ece85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl\" (UID: \"fb5edf49-030a-417f-9f33-ed2b091ece85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl"
Jul 17 22:29:18.241815 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.241797    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5edf49-030a-417f-9f33-ed2b091ece85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl\" (UID: \"fb5edf49-030a-417f-9f33-ed2b091ece85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl"
Jul 17 22:29:18.389889 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.389858    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl"
Jul 17 22:29:18.419452 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.419420    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:18.586350 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:18.586237    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl"]
Jul 17 22:29:18.588996 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:29:18.588964    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5edf49_030a_417f_9f33_ed2b091ece85.slice/crio-7db45232a7fe47ab2d7c6546f7e8326f88911f10997a930dc62ed74b3db33922 WatchSource:0}: Error finding container 7db45232a7fe47ab2d7c6546f7e8326f88911f10997a930dc62ed74b3db33922: Status 404 returned error can't find the container with id 7db45232a7fe47ab2d7c6546f7e8326f88911f10997a930dc62ed74b3db33922
Jul 17 22:29:19.090195 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:19.090049    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl" event={"ID":"fb5edf49-030a-417f-9f33-ed2b091ece85","Type":"ContainerStarted","Data":"581d7dad4d4becdb41d38745e6afe8bd1223347965d8693c15d855255889a90f"}
Jul 17 22:29:19.090195 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:19.090160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl" event={"ID":"fb5edf49-030a-417f-9f33-ed2b091ece85","Type":"ContainerStarted","Data":"7db45232a7fe47ab2d7c6546f7e8326f88911f10997a930dc62ed74b3db33922"}
Jul 17 22:29:19.092162 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:19.092135    2565 generic.go:356] "Generic (PLEG): container finished" podID="381b35f1-f23d-4fad-8887-875cedd82e76" containerID="af3395c29a2e98bc495ecb4af3de5e15926f1eb460e14207aaea4a1232c99cf1" exitCode=6
Jul 17 22:29:19.092284 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:19.092203    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c" event={"ID":"381b35f1-f23d-4fad-8887-875cedd82e76","Type":"ContainerDied","Data":"af3395c29a2e98bc495ecb4af3de5e15926f1eb460e14207aaea4a1232c99cf1"}
Jul 17 22:29:19.092284 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:19.092258    2565 scope.go:117] "RemoveContainer" containerID="9e55eb385b534fa0d37b8ef023f963dbd7541b8627a6fa4be457618f5f9b12c0"
Jul 17 22:29:19.092591 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:19.092575    2565 scope.go:117] "RemoveContainer" containerID="af3395c29a2e98bc495ecb4af3de5e15926f1eb460e14207aaea4a1232c99cf1"
Jul 17 22:29:19.092804 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:29:19.092784    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c_odh-ai-gateway-infra(381b35f1-f23d-4fad-8887-875cedd82e76)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c" podUID="381b35f1-f23d-4fad-8887-875cedd82e76"
Jul 17 22:29:19.108200 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:19.108147    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl" podStartSLOduration=1.108124784 podStartE2EDuration="1.108124784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:29:19.106334772 +0000 UTC m=+2622.495351439" watchObservedRunningTime="2026-07-17 22:29:19.108124784 +0000 UTC m=+2622.497141436"
Jul 17 22:29:20.098894 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:20.098860    2565 generic.go:356] "Generic (PLEG): container finished" podID="fb5edf49-030a-417f-9f33-ed2b091ece85" containerID="581d7dad4d4becdb41d38745e6afe8bd1223347965d8693c15d855255889a90f" exitCode=6
Jul 17 22:29:20.099365 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:20.098936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl" event={"ID":"fb5edf49-030a-417f-9f33-ed2b091ece85","Type":"ContainerDied","Data":"581d7dad4d4becdb41d38745e6afe8bd1223347965d8693c15d855255889a90f"}
Jul 17 22:29:20.099365 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:20.099167    2565 scope.go:117] "RemoveContainer" containerID="581d7dad4d4becdb41d38745e6afe8bd1223347965d8693c15d855255889a90f"
Jul 17 22:29:20.132945 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:20.132916    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c"]
Jul 17 22:29:20.279855 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:20.279828    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c"
Jul 17 22:29:20.362418 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:20.362323    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381b35f1-f23d-4fad-8887-875cedd82e76-maas-api-service-ca\") pod \"381b35f1-f23d-4fad-8887-875cedd82e76\" (UID: \"381b35f1-f23d-4fad-8887-875cedd82e76\") "
Jul 17 22:29:20.362780 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:20.362754    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381b35f1-f23d-4fad-8887-875cedd82e76-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "381b35f1-f23d-4fad-8887-875cedd82e76" (UID: "381b35f1-f23d-4fad-8887-875cedd82e76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:29:20.463872 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:20.463837    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381b35f1-f23d-4fad-8887-875cedd82e76-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:29:21.107835 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:21.107800    2565 generic.go:356] "Generic (PLEG): container finished" podID="fb5edf49-030a-417f-9f33-ed2b091ece85" containerID="ac0e157016f8cf86124e99b695a7d807d811d2f27b1e03de816ca7f4162f9555" exitCode=6
Jul 17 22:29:21.108324 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:21.107886    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl" event={"ID":"fb5edf49-030a-417f-9f33-ed2b091ece85","Type":"ContainerDied","Data":"ac0e157016f8cf86124e99b695a7d807d811d2f27b1e03de816ca7f4162f9555"}
Jul 17 22:29:21.108324 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:21.107949    2565 scope.go:117] "RemoveContainer" containerID="581d7dad4d4becdb41d38745e6afe8bd1223347965d8693c15d855255889a90f"
Jul 17 22:29:21.108324 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:21.108202    2565 scope.go:117] "RemoveContainer" containerID="ac0e157016f8cf86124e99b695a7d807d811d2f27b1e03de816ca7f4162f9555"
Jul 17 22:29:21.108484 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:29:21.108464    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl_odh-ai-gateway-infra(fb5edf49-030a-417f-9f33-ed2b091ece85)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl" podUID="fb5edf49-030a-417f-9f33-ed2b091ece85"
Jul 17 22:29:21.109340 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:21.109318    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c" event={"ID":"381b35f1-f23d-4fad-8887-875cedd82e76","Type":"ContainerDied","Data":"94f023d7bbc80cdb822e996f4ff6f3a41d9f83bc01747269b88455a7cf59adb8"}
Jul 17 22:29:21.109413 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:21.109341    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c"
Jul 17 22:29:21.119891 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:21.119872    2565 scope.go:117] "RemoveContainer" containerID="af3395c29a2e98bc495ecb4af3de5e15926f1eb460e14207aaea4a1232c99cf1"
Jul 17 22:29:21.138090 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:21.138035    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c"]
Jul 17 22:29:21.145695 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:21.145672    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hhb5c"]
Jul 17 22:29:21.182286 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:21.182260    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="381b35f1-f23d-4fad-8887-875cedd82e76" path="/var/lib/kubelet/pods/381b35f1-f23d-4fad-8887-875cedd82e76/volumes"
Jul 17 22:29:22.116961 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:22.116929    2565 scope.go:117] "RemoveContainer" containerID="ac0e157016f8cf86124e99b695a7d807d811d2f27b1e03de816ca7f4162f9555"
Jul 17 22:29:22.117420 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:29:22.117204    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl_odh-ai-gateway-infra(fb5edf49-030a-417f-9f33-ed2b091ece85)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl" podUID="fb5edf49-030a-417f-9f33-ed2b091ece85"
Jul 17 22:29:22.739060 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:22.739027    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:22.841564 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:22.841531    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:24.063205 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:24.063172    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:24.132467 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:24.132438    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:24.734484 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:24.734448    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4"]
Jul 17 22:29:24.734839 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:24.734826    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381b35f1-f23d-4fad-8887-875cedd82e76" containerName="revoke-keys"
Jul 17 22:29:24.734879 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:24.734841    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="381b35f1-f23d-4fad-8887-875cedd82e76" containerName="revoke-keys"
Jul 17 22:29:24.734879 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:24.734858    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381b35f1-f23d-4fad-8887-875cedd82e76" containerName="revoke-keys"
Jul 17 22:29:24.734879 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:24.734866    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="381b35f1-f23d-4fad-8887-875cedd82e76" containerName="revoke-keys"
Jul 17 22:29:24.734978 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:24.734930    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5f6f404-542c-4b2e-8b13-4df557abe0bc" containerName="istio-proxy"
Jul 17 22:29:24.734978 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:24.734942    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="381b35f1-f23d-4fad-8887-875cedd82e76" containerName="revoke-keys"
Jul 17 22:29:24.734978 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:24.734956    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="381b35f1-f23d-4fad-8887-875cedd82e76" containerName="revoke-keys"
Jul 17 22:29:24.739342 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:24.739325    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4"
Jul 17 22:29:24.745257 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:24.745230    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4"]
Jul 17 22:29:24.807285 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:24.807249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79efa69-602e-44e2-bc1a-419a76fcf826-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4\" (UID: \"a79efa69-602e-44e2-bc1a-419a76fcf826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4"
Jul 17 22:29:24.908743 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:24.908705    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79efa69-602e-44e2-bc1a-419a76fcf826-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4\" (UID: \"a79efa69-602e-44e2-bc1a-419a76fcf826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4"
Jul 17 22:29:24.909373 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:24.909354    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79efa69-602e-44e2-bc1a-419a76fcf826-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4\" (UID: \"a79efa69-602e-44e2-bc1a-419a76fcf826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4"
Jul 17 22:29:25.052273 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:25.052233    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4"
Jul 17 22:29:25.195780 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:25.195629    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4"]
Jul 17 22:29:25.198121 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:29:25.198094    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79efa69_602e_44e2_bc1a_419a76fcf826.slice/crio-e4ee66795a1e2fbb76f32d419a5e40098aa9b77eea16e57915f9ef67fc4a9685 WatchSource:0}: Error finding container e4ee66795a1e2fbb76f32d419a5e40098aa9b77eea16e57915f9ef67fc4a9685: Status 404 returned error can't find the container with id e4ee66795a1e2fbb76f32d419a5e40098aa9b77eea16e57915f9ef67fc4a9685
Jul 17 22:29:26.139752 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:26.139655    2565 generic.go:356] "Generic (PLEG): container finished" podID="a79efa69-602e-44e2-bc1a-419a76fcf826" containerID="94f50244a8e1639e8c7052629c986eb51336747b92204c833cef290e16e5643c" exitCode=6
Jul 17 22:29:26.139752 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:26.139711    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4" event={"ID":"a79efa69-602e-44e2-bc1a-419a76fcf826","Type":"ContainerDied","Data":"94f50244a8e1639e8c7052629c986eb51336747b92204c833cef290e16e5643c"}
Jul 17 22:29:26.139971 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:26.139783    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4" event={"ID":"a79efa69-602e-44e2-bc1a-419a76fcf826","Type":"ContainerStarted","Data":"e4ee66795a1e2fbb76f32d419a5e40098aa9b77eea16e57915f9ef67fc4a9685"}
Jul 17 22:29:26.139971 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:26.139961    2565 scope.go:117] "RemoveContainer" containerID="94f50244a8e1639e8c7052629c986eb51336747b92204c833cef290e16e5643c"
Jul 17 22:29:27.146811 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:27.146718    2565 generic.go:356] "Generic (PLEG): container finished" podID="a79efa69-602e-44e2-bc1a-419a76fcf826" containerID="cbc49637754a032242a120476b5f70616dfadb0577e83ff3acec189bdfbf7716" exitCode=6
Jul 17 22:29:27.146811 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:27.146765    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4" event={"ID":"a79efa69-602e-44e2-bc1a-419a76fcf826","Type":"ContainerDied","Data":"cbc49637754a032242a120476b5f70616dfadb0577e83ff3acec189bdfbf7716"}
Jul 17 22:29:27.146811 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:27.146814    2565 scope.go:117] "RemoveContainer" containerID="94f50244a8e1639e8c7052629c986eb51336747b92204c833cef290e16e5643c"
Jul 17 22:29:27.147412 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:27.147019    2565 scope.go:117] "RemoveContainer" containerID="cbc49637754a032242a120476b5f70616dfadb0577e83ff3acec189bdfbf7716"
Jul 17 22:29:27.147412 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:29:27.147326    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4_odh-ai-gateway-infra(a79efa69-602e-44e2-bc1a-419a76fcf826)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4" podUID="a79efa69-602e-44e2-bc1a-419a76fcf826"
Jul 17 22:29:28.153021 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:28.152996    2565 scope.go:117] "RemoveContainer" containerID="cbc49637754a032242a120476b5f70616dfadb0577e83ff3acec189bdfbf7716"
Jul 17 22:29:28.153472 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:29:28.153204    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4_odh-ai-gateway-infra(a79efa69-602e-44e2-bc1a-419a76fcf826)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4" podUID="a79efa69-602e-44e2-bc1a-419a76fcf826"
Jul 17 22:29:30.997962 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:30.997925    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:31.372032 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.371927    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"]
Jul 17 22:29:31.372320 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.372307    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381b35f1-f23d-4fad-8887-875cedd82e76" containerName="revoke-keys"
Jul 17 22:29:31.372361 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.372322    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="381b35f1-f23d-4fad-8887-875cedd82e76" containerName="revoke-keys"
Jul 17 22:29:31.372392 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.372374    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="381b35f1-f23d-4fad-8887-875cedd82e76" containerName="revoke-keys"
Jul 17 22:29:31.376816 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.376796    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.379367 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.379349    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-895e50b8-openshift-default-dockercfg-cz5k4\""
Jul 17 22:29:31.393249 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.393222    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"]
Jul 17 22:29:31.407862 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.407820    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:31.469666 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.469629    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istiod-ca-cert\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.470008 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.469976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-data\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.470279 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.470254    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-credential-socket\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.470420 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.470289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-workload-certs\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.470420 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.470325    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-envoy\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.470420 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.470365    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-podinfo\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.470420 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.470395    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-workload-socket\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.470639 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.470507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-token\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.470639 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.470533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns84l\" (UniqueName: \"kubernetes.io/projected/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-kube-api-access-ns84l\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.571992 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.571956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-credential-socket\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.571992 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.571993    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-workload-certs\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.572287 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.572016    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-envoy\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.572287 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.572051    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-podinfo\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.572287 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.572096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-workload-socket\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.572287 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.572166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-token\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.572498 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.572186    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns84l\" (UniqueName: \"kubernetes.io/projected/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-kube-api-access-ns84l\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.572498 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.572434    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-credential-socket\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.572498 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.572471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istiod-ca-cert\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.572655 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.572469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-workload-certs\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.572655 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.572533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-data\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.572747 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.572658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-workload-socket\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.572817 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.572800    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-data\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.573019 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.573002    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istiod-ca-cert\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.574590 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.574572    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-envoy\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.574957 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.574936    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-podinfo\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.583121 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.583095    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-token\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.584256 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.584235    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns84l\" (UniqueName: \"kubernetes.io/projected/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-kube-api-access-ns84l\") pod \"e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") " pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.592387 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.592360    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:31.688031 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.687938    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:31.833235 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.833204    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"]
Jul 17 22:29:31.836002 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:29:31.835976    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ff01ba_e9bd_464c_9ca5_f715fc2b291a.slice/crio-a649479fbd2419f76a30d7c552fa8179dd39cd2db6efbc34a585a9cd6ec491c5 WatchSource:0}: Error finding container a649479fbd2419f76a30d7c552fa8179dd39cd2db6efbc34a585a9cd6ec491c5: Status 404 returned error can't find the container with id a649479fbd2419f76a30d7c552fa8179dd39cd2db6efbc34a585a9cd6ec491c5
Jul 17 22:29:31.838172 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.838134    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:29:31.838229 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.838216    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:29:31.838268 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:31.838248    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:29:32.174223 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:32.174182    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj" event={"ID":"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a","Type":"ContainerStarted","Data":"c5a0b7076347523ed2ea40b433282a1fa14dcbbd208e1dbe65524f7d9693b9c7"}
Jul 17 22:29:32.174666 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:32.174246    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj" event={"ID":"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a","Type":"ContainerStarted","Data":"a649479fbd2419f76a30d7c552fa8179dd39cd2db6efbc34a585a9cd6ec491c5"}
Jul 17 22:29:32.196603 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:32.196545    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj" podStartSLOduration=1.196529251 podStartE2EDuration="1.196529251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:29:32.19532862 +0000 UTC m=+2635.584345273" watchObservedRunningTime="2026-07-17 22:29:32.196529251 +0000 UTC m=+2635.585545896"
Jul 17 22:29:32.688396 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:32.688356    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:32.693110 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:32.693083    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:33.176882 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:33.176845    2565 scope.go:117] "RemoveContainer" containerID="ac0e157016f8cf86124e99b695a7d807d811d2f27b1e03de816ca7f4162f9555"
Jul 17 22:29:33.182609 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:33.182584    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:33.182740 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:33.182670    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:34.156836 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:34.156740    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:34.186246 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:34.186209    2565 generic.go:356] "Generic (PLEG): container finished" podID="fb5edf49-030a-417f-9f33-ed2b091ece85" containerID="70f9113bfd4f2741ad1c292e37e7fb6be15bc6b87abe00014f3dfecc4022779a" exitCode=6
Jul 17 22:29:34.186668 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:34.186270    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl" event={"ID":"fb5edf49-030a-417f-9f33-ed2b091ece85","Type":"ContainerDied","Data":"70f9113bfd4f2741ad1c292e37e7fb6be15bc6b87abe00014f3dfecc4022779a"}
Jul 17 22:29:34.186668 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:34.186333    2565 scope.go:117] "RemoveContainer" containerID="ac0e157016f8cf86124e99b695a7d807d811d2f27b1e03de816ca7f4162f9555"
Jul 17 22:29:34.186807 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:34.186796    2565 scope.go:117] "RemoveContainer" containerID="70f9113bfd4f2741ad1c292e37e7fb6be15bc6b87abe00014f3dfecc4022779a"
Jul 17 22:29:34.187017 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:29:34.186997    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl_odh-ai-gateway-infra(fb5edf49-030a-417f-9f33-ed2b091ece85)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl" podUID="fb5edf49-030a-417f-9f33-ed2b091ece85"
Jul 17 22:29:34.251993 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:34.251961    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:34.357365 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:34.357334    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:34.481618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:34.481524    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:34.914865 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:34.914827    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:35.015820 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:35.015784    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:35.224887 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:35.224786    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl"]
Jul 17 22:29:35.370757 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:35.370726    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl"
Jul 17 22:29:35.508513 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:35.508479    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5edf49-030a-417f-9f33-ed2b091ece85-maas-api-service-ca\") pod \"fb5edf49-030a-417f-9f33-ed2b091ece85\" (UID: \"fb5edf49-030a-417f-9f33-ed2b091ece85\") "
Jul 17 22:29:35.508862 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:35.508839    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5edf49-030a-417f-9f33-ed2b091ece85-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fb5edf49-030a-417f-9f33-ed2b091ece85" (UID: "fb5edf49-030a-417f-9f33-ed2b091ece85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:29:35.610166 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:35.610119    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5edf49-030a-417f-9f33-ed2b091ece85-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:29:36.200319 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:36.200268    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl" event={"ID":"fb5edf49-030a-417f-9f33-ed2b091ece85","Type":"ContainerDied","Data":"7db45232a7fe47ab2d7c6546f7e8326f88911f10997a930dc62ed74b3db33922"}
Jul 17 22:29:36.200319 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:36.200301    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl"
Jul 17 22:29:36.200555 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:36.200345    2565 scope.go:117] "RemoveContainer" containerID="70f9113bfd4f2741ad1c292e37e7fb6be15bc6b87abe00014f3dfecc4022779a"
Jul 17 22:29:36.231593 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:36.231552    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl"]
Jul 17 22:29:36.246116 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:36.246083    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-tm8hl"]
Jul 17 22:29:37.183187 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:37.183156    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb5edf49-030a-417f-9f33-ed2b091ece85" path="/var/lib/kubelet/pods/fb5edf49-030a-417f-9f33-ed2b091ece85/volumes"
Jul 17 22:29:38.658672 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:38.658641    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:29:39.247464 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:39.247430    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:29:42.176886 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:42.176851    2565 scope.go:117] "RemoveContainer" containerID="cbc49637754a032242a120476b5f70616dfadb0577e83ff3acec189bdfbf7716"
Jul 17 22:29:43.237298 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:43.237264    2565 generic.go:356] "Generic (PLEG): container finished" podID="a79efa69-602e-44e2-bc1a-419a76fcf826" containerID="e00aacabf3dda78408f34fb890f88c0e4e38bc6e9e450ca7cb109e5186100eeb" exitCode=6
Jul 17 22:29:43.237712 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:43.237309    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4" event={"ID":"a79efa69-602e-44e2-bc1a-419a76fcf826","Type":"ContainerDied","Data":"e00aacabf3dda78408f34fb890f88c0e4e38bc6e9e450ca7cb109e5186100eeb"}
Jul 17 22:29:43.237712 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:43.237363    2565 scope.go:117] "RemoveContainer" containerID="cbc49637754a032242a120476b5f70616dfadb0577e83ff3acec189bdfbf7716"
Jul 17 22:29:43.237712 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:43.237632    2565 scope.go:117] "RemoveContainer" containerID="e00aacabf3dda78408f34fb890f88c0e4e38bc6e9e450ca7cb109e5186100eeb"
Jul 17 22:29:43.237916 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:29:43.237891    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4_odh-ai-gateway-infra(a79efa69-602e-44e2-bc1a-419a76fcf826)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4" podUID="a79efa69-602e-44e2-bc1a-419a76fcf826"
Jul 17 22:29:44.262158 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:44.262113    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4"]
Jul 17 22:29:44.450692 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:44.450667    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4"
Jul 17 22:29:44.499341 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:44.499306    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79efa69-602e-44e2-bc1a-419a76fcf826-maas-api-service-ca\") pod \"a79efa69-602e-44e2-bc1a-419a76fcf826\" (UID: \"a79efa69-602e-44e2-bc1a-419a76fcf826\") "
Jul 17 22:29:44.499658 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:44.499636    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79efa69-602e-44e2-bc1a-419a76fcf826-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a79efa69-602e-44e2-bc1a-419a76fcf826" (UID: "a79efa69-602e-44e2-bc1a-419a76fcf826"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:29:44.600535 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:44.600495    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79efa69-602e-44e2-bc1a-419a76fcf826-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:29:45.249829 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:45.249797    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4"
Jul 17 22:29:45.249829 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:45.249822    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4" event={"ID":"a79efa69-602e-44e2-bc1a-419a76fcf826","Type":"ContainerDied","Data":"e4ee66795a1e2fbb76f32d419a5e40098aa9b77eea16e57915f9ef67fc4a9685"}
Jul 17 22:29:45.250098 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:45.249897    2565 scope.go:117] "RemoveContainer" containerID="e00aacabf3dda78408f34fb890f88c0e4e38bc6e9e450ca7cb109e5186100eeb"
Jul 17 22:29:45.267051 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:45.267021    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4"]
Jul 17 22:29:45.274399 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:45.274372    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-fnsp4"]
Jul 17 22:29:46.314845 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.314807    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k"]
Jul 17 22:29:46.315381 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.315359    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5edf49-030a-417f-9f33-ed2b091ece85" containerName="revoke-keys"
Jul 17 22:29:46.315452 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.315385    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5edf49-030a-417f-9f33-ed2b091ece85" containerName="revoke-keys"
Jul 17 22:29:46.315452 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.315409    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a79efa69-602e-44e2-bc1a-419a76fcf826" containerName="revoke-keys"
Jul 17 22:29:46.315452 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.315418    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a79efa69-602e-44e2-bc1a-419a76fcf826" containerName="revoke-keys"
Jul 17 22:29:46.315452 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.315429    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a79efa69-602e-44e2-bc1a-419a76fcf826" containerName="revoke-keys"
Jul 17 22:29:46.315452 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.315437    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a79efa69-602e-44e2-bc1a-419a76fcf826" containerName="revoke-keys"
Jul 17 22:29:46.315452 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.315448    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5edf49-030a-417f-9f33-ed2b091ece85" containerName="revoke-keys"
Jul 17 22:29:46.315734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.315456    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5edf49-030a-417f-9f33-ed2b091ece85" containerName="revoke-keys"
Jul 17 22:29:46.315734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.315553    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5edf49-030a-417f-9f33-ed2b091ece85" containerName="revoke-keys"
Jul 17 22:29:46.315734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.315564    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5edf49-030a-417f-9f33-ed2b091ece85" containerName="revoke-keys"
Jul 17 22:29:46.315734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.315576    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a79efa69-602e-44e2-bc1a-419a76fcf826" containerName="revoke-keys"
Jul 17 22:29:46.315734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.315587    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a79efa69-602e-44e2-bc1a-419a76fcf826" containerName="revoke-keys"
Jul 17 22:29:46.315734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.315596    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a79efa69-602e-44e2-bc1a-419a76fcf826" containerName="revoke-keys"
Jul 17 22:29:46.321445 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.321422    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k"
Jul 17 22:29:46.324414 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.324390    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:29:46.324828 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.324806    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:29:46.325448 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.325426    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k"]
Jul 17 22:29:46.416341 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.416298    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b1287e-70f1-4201-b4c1-9f8df2cb79c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k\" (UID: \"42b1287e-70f1-4201-b4c1-9f8df2cb79c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k"
Jul 17 22:29:46.516976 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.516936    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b1287e-70f1-4201-b4c1-9f8df2cb79c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k\" (UID: \"42b1287e-70f1-4201-b4c1-9f8df2cb79c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k"
Jul 17 22:29:46.517586 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.517568    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b1287e-70f1-4201-b4c1-9f8df2cb79c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k\" (UID: \"42b1287e-70f1-4201-b4c1-9f8df2cb79c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k"
Jul 17 22:29:46.636527 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.636439    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k"
Jul 17 22:29:46.786261 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:46.786228    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k"]
Jul 17 22:29:46.790259 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:29:46.790214    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b1287e_70f1_4201_b4c1_9f8df2cb79c4.slice/crio-065067be91bd8d9bf8669d6dc2981c9550dba2e9bc670726ddc6e89ec5e6857e WatchSource:0}: Error finding container 065067be91bd8d9bf8669d6dc2981c9550dba2e9bc670726ddc6e89ec5e6857e: Status 404 returned error can't find the container with id 065067be91bd8d9bf8669d6dc2981c9550dba2e9bc670726ddc6e89ec5e6857e
Jul 17 22:29:47.183294 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:47.183259    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a79efa69-602e-44e2-bc1a-419a76fcf826" path="/var/lib/kubelet/pods/a79efa69-602e-44e2-bc1a-419a76fcf826/volumes"
Jul 17 22:29:47.266625 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:47.266582    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k" event={"ID":"42b1287e-70f1-4201-b4c1-9f8df2cb79c4","Type":"ContainerStarted","Data":"0598b89f087553c7d867ef230c865769ce6890a6859485fbee6ae151ff675be3"}
Jul 17 22:29:47.276371 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:47.266677    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k" event={"ID":"42b1287e-70f1-4201-b4c1-9f8df2cb79c4","Type":"ContainerStarted","Data":"065067be91bd8d9bf8669d6dc2981c9550dba2e9bc670726ddc6e89ec5e6857e"}
Jul 17 22:29:47.282715 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:47.282665    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k" podStartSLOduration=1.282651363 podStartE2EDuration="1.282651363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:29:47.282121224 +0000 UTC m=+2650.671137899" watchObservedRunningTime="2026-07-17 22:29:47.282651363 +0000 UTC m=+2650.671668013"
Jul 17 22:29:48.181529 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:48.181494    2565 patch_prober.go:28] interesting pod/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Jul 17 22:29:48.181920 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:48.181555    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj" podUID="e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Jul 17 22:29:48.273420 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:48.273388    2565 generic.go:356] "Generic (PLEG): container finished" podID="42b1287e-70f1-4201-b4c1-9f8df2cb79c4" containerID="0598b89f087553c7d867ef230c865769ce6890a6859485fbee6ae151ff675be3" exitCode=6
Jul 17 22:29:48.273611 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:48.273475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k" event={"ID":"42b1287e-70f1-4201-b4c1-9f8df2cb79c4","Type":"ContainerDied","Data":"0598b89f087553c7d867ef230c865769ce6890a6859485fbee6ae151ff675be3"}
Jul 17 22:29:48.273719 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:48.273703    2565 scope.go:117] "RemoveContainer" containerID="0598b89f087553c7d867ef230c865769ce6890a6859485fbee6ae151ff675be3"
Jul 17 22:29:48.275114 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:48.275091    2565 generic.go:356] "Generic (PLEG): container finished" podID="e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" containerID="c5a0b7076347523ed2ea40b433282a1fa14dcbbd208e1dbe65524f7d9693b9c7" exitCode=0
Jul 17 22:29:48.275222 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:48.275097    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj" event={"ID":"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a","Type":"ContainerDied","Data":"c5a0b7076347523ed2ea40b433282a1fa14dcbbd208e1dbe65524f7d9693b9c7"}
Jul 17 22:29:48.275435 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:48.275424    2565 scope.go:117] "RemoveContainer" containerID="c5a0b7076347523ed2ea40b433282a1fa14dcbbd208e1dbe65524f7d9693b9c7"
Jul 17 22:29:48.276362 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:48.276335    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:29:48.276424 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:48.276405    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:29:48.276460 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:48.276437    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:29:49.285122 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:49.285089    2565 generic.go:356] "Generic (PLEG): container finished" podID="42b1287e-70f1-4201-b4c1-9f8df2cb79c4" containerID="76152e9718ab88a7721ad7382e3efe20fcc8692ce3c7c2c5e75096893bf0e8f0" exitCode=6
Jul 17 22:29:49.285570 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:49.285166    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k" event={"ID":"42b1287e-70f1-4201-b4c1-9f8df2cb79c4","Type":"ContainerDied","Data":"76152e9718ab88a7721ad7382e3efe20fcc8692ce3c7c2c5e75096893bf0e8f0"}
Jul 17 22:29:49.285570 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:49.285233    2565 scope.go:117] "RemoveContainer" containerID="0598b89f087553c7d867ef230c865769ce6890a6859485fbee6ae151ff675be3"
Jul 17 22:29:49.285570 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:49.285462    2565 scope.go:117] "RemoveContainer" containerID="76152e9718ab88a7721ad7382e3efe20fcc8692ce3c7c2c5e75096893bf0e8f0"
Jul 17 22:29:49.285754 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:29:49.285723    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k_odh-ai-gateway-infra(42b1287e-70f1-4201-b4c1-9f8df2cb79c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k" podUID="42b1287e-70f1-4201-b4c1-9f8df2cb79c4"
Jul 17 22:29:49.287444 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:49.287423    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj" event={"ID":"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a","Type":"ContainerStarted","Data":"467e66763f44e560f12ad0290bcd9f35d7f9af598502b73aa656a13df0aed12c"}
Jul 17 22:29:49.689168 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:49.689048    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:49.689450 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:49.689421    2565 patch_prober.go:28] interesting pod/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Jul 17 22:29:49.689567 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:49.689472    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj" podUID="e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Jul 17 22:29:50.293808 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:50.293783    2565 scope.go:117] "RemoveContainer" containerID="76152e9718ab88a7721ad7382e3efe20fcc8692ce3c7c2c5e75096893bf0e8f0"
Jul 17 22:29:50.294227 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:29:50.293960    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k_odh-ai-gateway-infra(42b1287e-70f1-4201-b4c1-9f8df2cb79c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k" podUID="42b1287e-70f1-4201-b4c1-9f8df2cb79c4"
Jul 17 22:29:50.688853 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:50.688764    2565 patch_prober.go:28] interesting pod/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Jul 17 22:29:50.688853 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:50.688817    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj" podUID="e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Jul 17 22:29:51.689041 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:51.689000    2565 patch_prober.go:28] interesting pod/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Jul 17 22:29:51.689484 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:51.689093    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj" podUID="e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Jul 17 22:29:52.668701 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:52.668648    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96"]
Jul 17 22:29:52.669261 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:52.669232    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a79efa69-602e-44e2-bc1a-419a76fcf826" containerName="revoke-keys"
Jul 17 22:29:52.669387 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:52.669263    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a79efa69-602e-44e2-bc1a-419a76fcf826" containerName="revoke-keys"
Jul 17 22:29:52.669387 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:52.669305    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5edf49-030a-417f-9f33-ed2b091ece85" containerName="revoke-keys"
Jul 17 22:29:52.669387 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:52.669314    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5edf49-030a-417f-9f33-ed2b091ece85" containerName="revoke-keys"
Jul 17 22:29:52.669541 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:52.669400    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5edf49-030a-417f-9f33-ed2b091ece85" containerName="revoke-keys"
Jul 17 22:29:52.674571 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:52.674543    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96"
Jul 17 22:29:52.681954 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:52.681924    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96"]
Jul 17 22:29:52.689846 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:52.689817    2565 patch_prober.go:28] interesting pod/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Jul 17 22:29:52.690313 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:52.689912    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj" podUID="e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Jul 17 22:29:52.788812 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:52.788178    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d46a1d2-4d45-4cfa-aff4-3c4472468d71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-9sm96\" (UID: \"8d46a1d2-4d45-4cfa-aff4-3c4472468d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96"
Jul 17 22:29:52.889341 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:52.889302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d46a1d2-4d45-4cfa-aff4-3c4472468d71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-9sm96\" (UID: \"8d46a1d2-4d45-4cfa-aff4-3c4472468d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96"
Jul 17 22:29:52.889951 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:52.889929    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d46a1d2-4d45-4cfa-aff4-3c4472468d71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-9sm96\" (UID: \"8d46a1d2-4d45-4cfa-aff4-3c4472468d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96"
Jul 17 22:29:52.991013 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:52.990914    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96"
Jul 17 22:29:53.341421 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:53.341389    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96"]
Jul 17 22:29:53.344016 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:29:53.343985    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d46a1d2_4d45_4cfa_aff4_3c4472468d71.slice/crio-f62b7bc8bc8a7cdf4dd86c0bb96e5e3890d1b607188a3931a7bc5141a02c948a WatchSource:0}: Error finding container f62b7bc8bc8a7cdf4dd86c0bb96e5e3890d1b607188a3931a7bc5141a02c948a: Status 404 returned error can't find the container with id f62b7bc8bc8a7cdf4dd86c0bb96e5e3890d1b607188a3931a7bc5141a02c948a
Jul 17 22:29:53.693450 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:53.693420    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:53.693814 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:53.693773    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:53.694500 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:53.694468    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:29:54.314008 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:54.313908    2565 generic.go:356] "Generic (PLEG): container finished" podID="8d46a1d2-4d45-4cfa-aff4-3c4472468d71" containerID="850791b4cfb9e99cbdae6182550e44601ba83bbdd655f9d9c3471a4ef5660701" exitCode=6
Jul 17 22:29:54.314225 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:54.314008    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96" event={"ID":"8d46a1d2-4d45-4cfa-aff4-3c4472468d71","Type":"ContainerDied","Data":"850791b4cfb9e99cbdae6182550e44601ba83bbdd655f9d9c3471a4ef5660701"}
Jul 17 22:29:54.314225 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:54.314105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96" event={"ID":"8d46a1d2-4d45-4cfa-aff4-3c4472468d71","Type":"ContainerStarted","Data":"f62b7bc8bc8a7cdf4dd86c0bb96e5e3890d1b607188a3931a7bc5141a02c948a"}
Jul 17 22:29:54.314225 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:54.314211    2565 scope.go:117] "RemoveContainer" containerID="850791b4cfb9e99cbdae6182550e44601ba83bbdd655f9d9c3471a4ef5660701"
Jul 17 22:29:55.321763 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:55.321727    2565 generic.go:356] "Generic (PLEG): container finished" podID="8d46a1d2-4d45-4cfa-aff4-3c4472468d71" containerID="a8c6eec520882543f84d4ae79a8bc4145b18df3cca41df731f953b086b3a90b6" exitCode=6
Jul 17 22:29:55.322288 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:55.321836    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96" event={"ID":"8d46a1d2-4d45-4cfa-aff4-3c4472468d71","Type":"ContainerDied","Data":"a8c6eec520882543f84d4ae79a8bc4145b18df3cca41df731f953b086b3a90b6"}
Jul 17 22:29:55.322288 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:55.321932    2565 scope.go:117] "RemoveContainer" containerID="850791b4cfb9e99cbdae6182550e44601ba83bbdd655f9d9c3471a4ef5660701"
Jul 17 22:29:55.322288 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:55.322231    2565 scope.go:117] "RemoveContainer" containerID="a8c6eec520882543f84d4ae79a8bc4145b18df3cca41df731f953b086b3a90b6"
Jul 17 22:29:55.322490 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:29:55.322464    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-9sm96_odh-ai-gateway-infra(8d46a1d2-4d45-4cfa-aff4-3c4472468d71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96" podUID="8d46a1d2-4d45-4cfa-aff4-3c4472468d71"
Jul 17 22:29:56.328798 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:29:56.328766    2565 scope.go:117] "RemoveContainer" containerID="a8c6eec520882543f84d4ae79a8bc4145b18df3cca41df731f953b086b3a90b6"
Jul 17 22:29:56.329230 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:29:56.328967    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-9sm96_odh-ai-gateway-infra(8d46a1d2-4d45-4cfa-aff4-3c4472468d71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96" podUID="8d46a1d2-4d45-4cfa-aff4-3c4472468d71"
Jul 17 22:30:00.138844 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:00.138765    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738790-7xgkf"]
Jul 17 22:30:00.144410 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:00.144390    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-895e50b8-29738790-kfbvx"]
Jul 17 22:30:00.144533 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:00.144514    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738790-7xgkf"
Jul 17 22:30:00.147901 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:00.147883    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-895e50b8-29738790-kfbvx"
Jul 17 22:30:00.155931 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:00.155906    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738790-7xgkf"
Jul 17 22:30:00.156058 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:00.156012    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738790-7xgkf"]
Jul 17 22:30:00.160674 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:00.160650    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-895e50b8-29738790-kfbvx"]
Jul 17 22:30:00.164575 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:00.164554    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-895e50b8-29738790-kfbvx"
Jul 17 22:30:00.316364 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:00.316326    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:30:00.336258 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:00.336225    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738790-7xgkf"]
Jul 17 22:30:00.339098 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:30:00.339044    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd051730a_986a_4be3_befe_03a1470b857d.slice/crio-929bd2168c51fb2f70273d85408862a0af2866742d5dfdbfae131848517d469b WatchSource:0}: Error finding container 929bd2168c51fb2f70273d85408862a0af2866742d5dfdbfae131848517d469b: Status 404 returned error can't find the container with id 929bd2168c51fb2f70273d85408862a0af2866742d5dfdbfae131848517d469b
Jul 17 22:30:00.348393 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:00.348365    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738790-7xgkf" event={"ID":"d051730a-986a-4be3-befe-03a1470b857d","Type":"ContainerStarted","Data":"929bd2168c51fb2f70273d85408862a0af2866742d5dfdbfae131848517d469b"}
Jul 17 22:30:00.356875 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:00.356854    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-895e50b8-29738790-kfbvx"]
Jul 17 22:30:01.355211 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:01.355174    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738790-7xgkf" event={"ID":"d051730a-986a-4be3-befe-03a1470b857d","Type":"ContainerStarted","Data":"914dd9b0b43ae57509d26ac57283f95f90876676a4cf7e6312b98421edfe18a1"}
Jul 17 22:30:01.356835 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:01.356796    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-895e50b8-29738790-kfbvx" event={"ID":"9cc61896-3906-412a-b6d6-ea91a856973d","Type":"ContainerStarted","Data":"ee626e7a27406b2de8de632a8df3bd832d930cd64ea15f36634b1e4862d6410f"}
Jul 17 22:30:01.356998 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:01.356862    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-895e50b8-29738790-kfbvx" event={"ID":"9cc61896-3906-412a-b6d6-ea91a856973d","Type":"ContainerStarted","Data":"9af3e32fa3e4db773758ed9e473d0a19e70332c42b56fd43859a263799422c2f"}
Jul 17 22:30:01.393123 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:01.393046    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-895e50b8-29738790-kfbvx" podStartSLOduration=1.393029706 podStartE2EDuration="1.393029706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:30:01.391149077 +0000 UTC m=+2664.780165730" watchObservedRunningTime="2026-07-17 22:30:01.393029706 +0000 UTC m=+2664.782046356"
Jul 17 22:30:01.393809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:01.393781    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738790-7xgkf" podStartSLOduration=1.3937712979999999 podStartE2EDuration="1.393771298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:30:01.373599551 +0000 UTC m=+2664.762616202" watchObservedRunningTime="2026-07-17 22:30:01.393771298 +0000 UTC m=+2664.782787949"
Jul 17 22:30:02.364596 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:02.364553    2565 generic.go:356] "Generic (PLEG): container finished" podID="d051730a-986a-4be3-befe-03a1470b857d" containerID="914dd9b0b43ae57509d26ac57283f95f90876676a4cf7e6312b98421edfe18a1" exitCode=0
Jul 17 22:30:02.365112 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:02.364660    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738790-7xgkf" event={"ID":"d051730a-986a-4be3-befe-03a1470b857d","Type":"ContainerDied","Data":"914dd9b0b43ae57509d26ac57283f95f90876676a4cf7e6312b98421edfe18a1"}
Jul 17 22:30:02.366457 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:02.366430    2565 generic.go:356] "Generic (PLEG): container finished" podID="9cc61896-3906-412a-b6d6-ea91a856973d" containerID="ee626e7a27406b2de8de632a8df3bd832d930cd64ea15f36634b1e4862d6410f" exitCode=0
Jul 17 22:30:02.366558 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:02.366489    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-895e50b8-29738790-kfbvx" event={"ID":"9cc61896-3906-412a-b6d6-ea91a856973d","Type":"ContainerDied","Data":"ee626e7a27406b2de8de632a8df3bd832d930cd64ea15f36634b1e4862d6410f"}
Jul 17 22:30:03.176926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:03.176889    2565 scope.go:117] "RemoveContainer" containerID="76152e9718ab88a7721ad7382e3efe20fcc8692ce3c7c2c5e75096893bf0e8f0"
Jul 17 22:30:03.511243 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:03.511207    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-895e50b8-29738790-kfbvx"
Jul 17 22:30:03.548891 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:03.548862    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738790-7xgkf"
Jul 17 22:30:03.637163 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:03.637127    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:30:04.377476 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:04.377432    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738790-7xgkf"
Jul 17 22:30:04.377636 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:04.377434    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738790-7xgkf" event={"ID":"d051730a-986a-4be3-befe-03a1470b857d","Type":"ContainerDied","Data":"929bd2168c51fb2f70273d85408862a0af2866742d5dfdbfae131848517d469b"}
Jul 17 22:30:04.377636 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:04.377574    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="929bd2168c51fb2f70273d85408862a0af2866742d5dfdbfae131848517d469b"
Jul 17 22:30:04.378784 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:04.378761    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-895e50b8-29738790-kfbvx"
Jul 17 22:30:04.378784 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:04.378770    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-895e50b8-29738790-kfbvx" event={"ID":"9cc61896-3906-412a-b6d6-ea91a856973d","Type":"ContainerDied","Data":"9af3e32fa3e4db773758ed9e473d0a19e70332c42b56fd43859a263799422c2f"}
Jul 17 22:30:04.378948 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:04.378808    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9af3e32fa3e4db773758ed9e473d0a19e70332c42b56fd43859a263799422c2f"
Jul 17 22:30:04.380708 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:04.380689    2565 generic.go:356] "Generic (PLEG): container finished" podID="42b1287e-70f1-4201-b4c1-9f8df2cb79c4" containerID="15a50a748f4e609fbc9b75240740bbaff216f54b2b28d0279b72ecec7e138d13" exitCode=6
Jul 17 22:30:04.380796 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:04.380760    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k" event={"ID":"42b1287e-70f1-4201-b4c1-9f8df2cb79c4","Type":"ContainerDied","Data":"15a50a748f4e609fbc9b75240740bbaff216f54b2b28d0279b72ecec7e138d13"}
Jul 17 22:30:04.380844 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:04.380797    2565 scope.go:117] "RemoveContainer" containerID="76152e9718ab88a7721ad7382e3efe20fcc8692ce3c7c2c5e75096893bf0e8f0"
Jul 17 22:30:04.381140 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:04.381122    2565 scope.go:117] "RemoveContainer" containerID="15a50a748f4e609fbc9b75240740bbaff216f54b2b28d0279b72ecec7e138d13"
Jul 17 22:30:04.381343 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:30:04.381323    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k_odh-ai-gateway-infra(42b1287e-70f1-4201-b4c1-9f8df2cb79c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k" podUID="42b1287e-70f1-4201-b4c1-9f8df2cb79c4"
Jul 17 22:30:05.405429 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:05.405386    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k"]
Jul 17 22:30:05.552881 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:05.552857    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k"
Jul 17 22:30:05.613705 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:05.613667    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b1287e-70f1-4201-b4c1-9f8df2cb79c4-maas-api-service-ca\") pod \"42b1287e-70f1-4201-b4c1-9f8df2cb79c4\" (UID: \"42b1287e-70f1-4201-b4c1-9f8df2cb79c4\") "
Jul 17 22:30:05.614016 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:05.613986    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b1287e-70f1-4201-b4c1-9f8df2cb79c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42b1287e-70f1-4201-b4c1-9f8df2cb79c4" (UID: "42b1287e-70f1-4201-b4c1-9f8df2cb79c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:30:05.714809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:05.714714    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b1287e-70f1-4201-b4c1-9f8df2cb79c4-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:30:06.393771 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:06.393738    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k"
Jul 17 22:30:06.393962 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:06.393774    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k" event={"ID":"42b1287e-70f1-4201-b4c1-9f8df2cb79c4","Type":"ContainerDied","Data":"065067be91bd8d9bf8669d6dc2981c9550dba2e9bc670726ddc6e89ec5e6857e"}
Jul 17 22:30:06.393962 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:06.393841    2565 scope.go:117] "RemoveContainer" containerID="15a50a748f4e609fbc9b75240740bbaff216f54b2b28d0279b72ecec7e138d13"
Jul 17 22:30:06.415884 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:06.415853    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k"]
Jul 17 22:30:06.424001 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:06.423976    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-r8c2k"]
Jul 17 22:30:07.182799 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:07.182769    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42b1287e-70f1-4201-b4c1-9f8df2cb79c4" path="/var/lib/kubelet/pods/42b1287e-70f1-4201-b4c1-9f8df2cb79c4/volumes"
Jul 17 22:30:08.109013 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.108972    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq"]
Jul 17 22:30:08.109423 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.109375    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cc61896-3906-412a-b6d6-ea91a856973d" containerName="cleanup"
Jul 17 22:30:08.109423 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.109388    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cc61896-3906-412a-b6d6-ea91a856973d" containerName="cleanup"
Jul 17 22:30:08.109423 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.109406    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d051730a-986a-4be3-befe-03a1470b857d" containerName="cleanup"
Jul 17 22:30:08.109423 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.109411    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d051730a-986a-4be3-befe-03a1470b857d" containerName="cleanup"
Jul 17 22:30:08.109423 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.109420    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b1287e-70f1-4201-b4c1-9f8df2cb79c4" containerName="revoke-keys"
Jul 17 22:30:08.109423 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.109425    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b1287e-70f1-4201-b4c1-9f8df2cb79c4" containerName="revoke-keys"
Jul 17 22:30:08.109603 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.109436    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b1287e-70f1-4201-b4c1-9f8df2cb79c4" containerName="revoke-keys"
Jul 17 22:30:08.109603 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.109441    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b1287e-70f1-4201-b4c1-9f8df2cb79c4" containerName="revoke-keys"
Jul 17 22:30:08.109603 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.109491    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b1287e-70f1-4201-b4c1-9f8df2cb79c4" containerName="revoke-keys"
Jul 17 22:30:08.109603 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.109500    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cc61896-3906-412a-b6d6-ea91a856973d" containerName="cleanup"
Jul 17 22:30:08.109603 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.109507    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d051730a-986a-4be3-befe-03a1470b857d" containerName="cleanup"
Jul 17 22:30:08.109603 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.109514    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b1287e-70f1-4201-b4c1-9f8df2cb79c4" containerName="revoke-keys"
Jul 17 22:30:08.113815 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.113797    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq"
Jul 17 22:30:08.119511 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.119484    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq"]
Jul 17 22:30:08.176807 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.176780    2565 scope.go:117] "RemoveContainer" containerID="a8c6eec520882543f84d4ae79a8bc4145b18df3cca41df731f953b086b3a90b6"
Jul 17 22:30:08.239254 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.239222    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq\" (UID: \"a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq"
Jul 17 22:30:08.339926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.339886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq\" (UID: \"a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq"
Jul 17 22:30:08.340549 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.340523    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq\" (UID: \"a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq"
Jul 17 22:30:08.425235 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.425204    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq"
Jul 17 22:30:08.585746 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:08.585718    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq"]
Jul 17 22:30:08.587874 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:30:08.587843    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8809ed1_c9d7_4c50_828b_f5f8dadeb5fc.slice/crio-c5af92231cf2dab42c33e0074640617671e9e762deb054762481f58693731533 WatchSource:0}: Error finding container c5af92231cf2dab42c33e0074640617671e9e762deb054762481f58693731533: Status 404 returned error can't find the container with id c5af92231cf2dab42c33e0074640617671e9e762deb054762481f58693731533
Jul 17 22:30:09.410609 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:09.410573    2565 generic.go:356] "Generic (PLEG): container finished" podID="a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc" containerID="697184842ed8b10476bb6634f51086b9453fd14aaa4db9cf25664a9746fff76d" exitCode=6
Jul 17 22:30:09.411042 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:09.410682    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq" event={"ID":"a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc","Type":"ContainerDied","Data":"697184842ed8b10476bb6634f51086b9453fd14aaa4db9cf25664a9746fff76d"}
Jul 17 22:30:09.411042 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:09.410727    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq" event={"ID":"a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc","Type":"ContainerStarted","Data":"c5af92231cf2dab42c33e0074640617671e9e762deb054762481f58693731533"}
Jul 17 22:30:09.411042 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:09.410796    2565 scope.go:117] "RemoveContainer" containerID="697184842ed8b10476bb6634f51086b9453fd14aaa4db9cf25664a9746fff76d"
Jul 17 22:30:09.412591 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:09.412567    2565 generic.go:356] "Generic (PLEG): container finished" podID="8d46a1d2-4d45-4cfa-aff4-3c4472468d71" containerID="f74d595e02a4f544a30d5aea0056b637fa44619db0a81582a6564b2e6247e2e8" exitCode=6
Jul 17 22:30:09.412692 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:09.412636    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96" event={"ID":"8d46a1d2-4d45-4cfa-aff4-3c4472468d71","Type":"ContainerDied","Data":"f74d595e02a4f544a30d5aea0056b637fa44619db0a81582a6564b2e6247e2e8"}
Jul 17 22:30:09.412742 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:09.412691    2565 scope.go:117] "RemoveContainer" containerID="a8c6eec520882543f84d4ae79a8bc4145b18df3cca41df731f953b086b3a90b6"
Jul 17 22:30:09.412948 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:09.412932    2565 scope.go:117] "RemoveContainer" containerID="f74d595e02a4f544a30d5aea0056b637fa44619db0a81582a6564b2e6247e2e8"
Jul 17 22:30:09.413211 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:30:09.413176    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-9sm96_odh-ai-gateway-infra(8d46a1d2-4d45-4cfa-aff4-3c4472468d71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96" podUID="8d46a1d2-4d45-4cfa-aff4-3c4472468d71"
Jul 17 22:30:10.419214 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:10.419178    2565 generic.go:356] "Generic (PLEG): container finished" podID="a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc" containerID="a008864ef9330cd64af3b3b421639d662eed4f8666a6bb29b35912b7d42e40cd" exitCode=6
Jul 17 22:30:10.419694 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:10.419260    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq" event={"ID":"a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc","Type":"ContainerDied","Data":"a008864ef9330cd64af3b3b421639d662eed4f8666a6bb29b35912b7d42e40cd"}
Jul 17 22:30:10.419694 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:10.419324    2565 scope.go:117] "RemoveContainer" containerID="697184842ed8b10476bb6634f51086b9453fd14aaa4db9cf25664a9746fff76d"
Jul 17 22:30:10.419694 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:10.419502    2565 scope.go:117] "RemoveContainer" containerID="a008864ef9330cd64af3b3b421639d662eed4f8666a6bb29b35912b7d42e40cd"
Jul 17 22:30:10.419847 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:30:10.419725    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq_odh-ai-gateway-infra(a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq" podUID="a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc"
Jul 17 22:30:10.450107 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:10.450058    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96"]
Jul 17 22:30:10.584929 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:10.584902    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96"
Jul 17 22:30:10.659386 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:10.659348    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d46a1d2-4d45-4cfa-aff4-3c4472468d71-maas-api-service-ca\") pod \"8d46a1d2-4d45-4cfa-aff4-3c4472468d71\" (UID: \"8d46a1d2-4d45-4cfa-aff4-3c4472468d71\") "
Jul 17 22:30:10.659755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:10.659728    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d46a1d2-4d45-4cfa-aff4-3c4472468d71-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8d46a1d2-4d45-4cfa-aff4-3c4472468d71" (UID: "8d46a1d2-4d45-4cfa-aff4-3c4472468d71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:30:10.760698 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:10.760664    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d46a1d2-4d45-4cfa-aff4-3c4472468d71-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:30:11.427851 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:11.427820    2565 scope.go:117] "RemoveContainer" containerID="a008864ef9330cd64af3b3b421639d662eed4f8666a6bb29b35912b7d42e40cd"
Jul 17 22:30:11.428317 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:30:11.428123    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq_odh-ai-gateway-infra(a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq" podUID="a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc"
Jul 17 22:30:11.428876 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:11.428854    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96" event={"ID":"8d46a1d2-4d45-4cfa-aff4-3c4472468d71","Type":"ContainerDied","Data":"f62b7bc8bc8a7cdf4dd86c0bb96e5e3890d1b607188a3931a7bc5141a02c948a"}
Jul 17 22:30:11.428994 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:11.428912    2565 scope.go:117] "RemoveContainer" containerID="f74d595e02a4f544a30d5aea0056b637fa44619db0a81582a6564b2e6247e2e8"
Jul 17 22:30:11.428994 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:11.428861    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96"
Jul 17 22:30:11.454784 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:11.454697    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96"]
Jul 17 22:30:11.458801 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:11.458773    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-9sm96"]
Jul 17 22:30:13.182436 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:13.182398    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d46a1d2-4d45-4cfa-aff4-3c4472468d71" path="/var/lib/kubelet/pods/8d46a1d2-4d45-4cfa-aff4-3c4472468d71/volumes"
Jul 17 22:30:18.648941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:18.648901    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:30:22.044307 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.044266    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-895e50b8-nj754"]
Jul 17 22:30:22.044879 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.044785    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d46a1d2-4d45-4cfa-aff4-3c4472468d71" containerName="revoke-keys"
Jul 17 22:30:22.044879 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.044805    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d46a1d2-4d45-4cfa-aff4-3c4472468d71" containerName="revoke-keys"
Jul 17 22:30:22.044879 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.044827    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b1287e-70f1-4201-b4c1-9f8df2cb79c4" containerName="revoke-keys"
Jul 17 22:30:22.044879 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.044836    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b1287e-70f1-4201-b4c1-9f8df2cb79c4" containerName="revoke-keys"
Jul 17 22:30:22.044879 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.044859    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d46a1d2-4d45-4cfa-aff4-3c4472468d71" containerName="revoke-keys"
Jul 17 22:30:22.044879 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.044868    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d46a1d2-4d45-4cfa-aff4-3c4472468d71" containerName="revoke-keys"
Jul 17 22:30:22.044879 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.044883    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d46a1d2-4d45-4cfa-aff4-3c4472468d71" containerName="revoke-keys"
Jul 17 22:30:22.045258 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.044890    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d46a1d2-4d45-4cfa-aff4-3c4472468d71" containerName="revoke-keys"
Jul 17 22:30:22.045258 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.044982    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d46a1d2-4d45-4cfa-aff4-3c4472468d71" containerName="revoke-keys"
Jul 17 22:30:22.045258 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.044993    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d46a1d2-4d45-4cfa-aff4-3c4472468d71" containerName="revoke-keys"
Jul 17 22:30:22.045258 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.045002    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b1287e-70f1-4201-b4c1-9f8df2cb79c4" containerName="revoke-keys"
Jul 17 22:30:22.048374 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.048352    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-895e50b8-nj754"
Jul 17 22:30:22.055424 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.055392    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-895e50b8-nj754"]
Jul 17 22:30:22.170405 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.170368    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f21a49-3718-4779-b42b-938bf462ba74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-895e50b8-nj754\" (UID: \"69f21a49-3718-4779-b42b-938bf462ba74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-895e50b8-nj754"
Jul 17 22:30:22.271752 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.271711    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f21a49-3718-4779-b42b-938bf462ba74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-895e50b8-nj754\" (UID: \"69f21a49-3718-4779-b42b-938bf462ba74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-895e50b8-nj754"
Jul 17 22:30:22.272444 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.272422    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f21a49-3718-4779-b42b-938bf462ba74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-895e50b8-nj754\" (UID: \"69f21a49-3718-4779-b42b-938bf462ba74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-895e50b8-nj754"
Jul 17 22:30:22.360592 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.360483    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-895e50b8-nj754"
Jul 17 22:30:22.498442 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:22.498406    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-895e50b8-nj754"]
Jul 17 22:30:22.499502 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:30:22.499469    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f21a49_3718_4779_b42b_938bf462ba74.slice/crio-e0697edb92216eae60a1396fa536fe2753678b59b37df896736f9f3855c8bf7c WatchSource:0}: Error finding container e0697edb92216eae60a1396fa536fe2753678b59b37df896736f9f3855c8bf7c: Status 404 returned error can't find the container with id e0697edb92216eae60a1396fa536fe2753678b59b37df896736f9f3855c8bf7c
Jul 17 22:30:23.491356 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:23.491252    2565 generic.go:356] "Generic (PLEG): container finished" podID="69f21a49-3718-4779-b42b-938bf462ba74" containerID="858a1ff6721560848e958a2c9786afa4b917e8af88054407c8816a490162b70a" exitCode=0
Jul 17 22:30:23.491740 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:23.491347    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-895e50b8-nj754" event={"ID":"69f21a49-3718-4779-b42b-938bf462ba74","Type":"ContainerDied","Data":"858a1ff6721560848e958a2c9786afa4b917e8af88054407c8816a490162b70a"}
Jul 17 22:30:23.491740 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:23.491411    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-895e50b8-nj754" event={"ID":"69f21a49-3718-4779-b42b-938bf462ba74","Type":"ContainerStarted","Data":"e0697edb92216eae60a1396fa536fe2753678b59b37df896736f9f3855c8bf7c"}
Jul 17 22:30:24.176468 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.176426    2565 scope.go:117] "RemoveContainer" containerID="a008864ef9330cd64af3b3b421639d662eed4f8666a6bb29b35912b7d42e40cd"
Jul 17 22:30:24.497852 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.497814    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq" event={"ID":"a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc","Type":"ContainerStarted","Data":"79a022b5d204df67b7bbcf5b3990794cd8dd152ffdf75ab063da86d37ec791c5"}
Jul 17 22:30:24.513538 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.513488    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq" podStartSLOduration=16.513470921 podStartE2EDuration="16.513470921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:30:24.511758884 +0000 UTC m=+2687.900775535" watchObservedRunningTime="2026-07-17 22:30:24.513470921 +0000 UTC m=+2687.902487573"
Jul 17 22:30:24.631663 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.631637    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-895e50b8-nj754"
Jul 17 22:30:24.775006 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.774964    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh"]
Jul 17 22:30:24.775383 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.775369    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69f21a49-3718-4779-b42b-938bf462ba74" containerName="revoke-keys"
Jul 17 22:30:24.775438 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.775384    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="69f21a49-3718-4779-b42b-938bf462ba74" containerName="revoke-keys"
Jul 17 22:30:24.775478 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.775442    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d46a1d2-4d45-4cfa-aff4-3c4472468d71" containerName="revoke-keys"
Jul 17 22:30:24.775478 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.775453    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="69f21a49-3718-4779-b42b-938bf462ba74" containerName="revoke-keys"
Jul 17 22:30:24.778422 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.778405    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh"
Jul 17 22:30:24.785007 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.784972    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh"]
Jul 17 22:30:24.795589 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.795558    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f21a49-3718-4779-b42b-938bf462ba74-maas-api-service-ca\") pod \"69f21a49-3718-4779-b42b-938bf462ba74\" (UID: \"69f21a49-3718-4779-b42b-938bf462ba74\") "
Jul 17 22:30:24.795977 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.795957    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f21a49-3718-4779-b42b-938bf462ba74-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69f21a49-3718-4779-b42b-938bf462ba74" (UID: "69f21a49-3718-4779-b42b-938bf462ba74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:30:24.897237 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.897193    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d51a5b0-fc3c-42f0-9ace-cf914878b659-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh\" (UID: \"5d51a5b0-fc3c-42f0-9ace-cf914878b659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh"
Jul 17 22:30:24.897422 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.897377    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69f21a49-3718-4779-b42b-938bf462ba74-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:30:24.997993 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.997951    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d51a5b0-fc3c-42f0-9ace-cf914878b659-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh\" (UID: \"5d51a5b0-fc3c-42f0-9ace-cf914878b659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh"
Jul 17 22:30:24.998658 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:24.998634    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d51a5b0-fc3c-42f0-9ace-cf914878b659-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh\" (UID: \"5d51a5b0-fc3c-42f0-9ace-cf914878b659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh"
Jul 17 22:30:25.090460 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:25.090361    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh"
Jul 17 22:30:25.232238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:25.232205    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh"]
Jul 17 22:30:25.233879 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:30:25.233853    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d51a5b0_fc3c_42f0_9ace_cf914878b659.slice/crio-2c4f00d18e8e0773745d4b4f5e8134e7a2a7f94ecefe8fe7044e4fa08f887059 WatchSource:0}: Error finding container 2c4f00d18e8e0773745d4b4f5e8134e7a2a7f94ecefe8fe7044e4fa08f887059: Status 404 returned error can't find the container with id 2c4f00d18e8e0773745d4b4f5e8134e7a2a7f94ecefe8fe7044e4fa08f887059
Jul 17 22:30:25.503454 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:25.503410    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh" event={"ID":"5d51a5b0-fc3c-42f0-9ace-cf914878b659","Type":"ContainerStarted","Data":"2c4f00d18e8e0773745d4b4f5e8134e7a2a7f94ecefe8fe7044e4fa08f887059"}
Jul 17 22:30:25.504674 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:25.504645    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-895e50b8-nj754"
Jul 17 22:30:25.504674 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:25.504661    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-895e50b8-nj754" event={"ID":"69f21a49-3718-4779-b42b-938bf462ba74","Type":"ContainerDied","Data":"e0697edb92216eae60a1396fa536fe2753678b59b37df896736f9f3855c8bf7c"}
Jul 17 22:30:25.504949 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:25.504704    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0697edb92216eae60a1396fa536fe2753678b59b37df896736f9f3855c8bf7c"
Jul 17 22:30:25.506653 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:25.506622    2565 generic.go:356] "Generic (PLEG): container finished" podID="a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc" containerID="79a022b5d204df67b7bbcf5b3990794cd8dd152ffdf75ab063da86d37ec791c5" exitCode=6
Jul 17 22:30:25.506769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:25.506648    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq" event={"ID":"a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc","Type":"ContainerDied","Data":"79a022b5d204df67b7bbcf5b3990794cd8dd152ffdf75ab063da86d37ec791c5"}
Jul 17 22:30:25.506769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:25.506713    2565 scope.go:117] "RemoveContainer" containerID="a008864ef9330cd64af3b3b421639d662eed4f8666a6bb29b35912b7d42e40cd"
Jul 17 22:30:25.507123 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:25.507090    2565 scope.go:117] "RemoveContainer" containerID="79a022b5d204df67b7bbcf5b3990794cd8dd152ffdf75ab063da86d37ec791c5"
Jul 17 22:30:25.507352 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:30:25.507326    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq_odh-ai-gateway-infra(a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq" podUID="a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc"
Jul 17 22:30:25.522502 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:25.522474    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq"]
Jul 17 22:30:26.514458 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:26.514426    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d51a5b0-fc3c-42f0-9ace-cf914878b659" containerID="1982bc0efd497620ccb817d3c15e3f27ef07bd78c040ae70c686293b22bebe84" exitCode=6
Jul 17 22:30:26.514784 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:26.514467    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh" event={"ID":"5d51a5b0-fc3c-42f0-9ace-cf914878b659","Type":"ContainerDied","Data":"1982bc0efd497620ccb817d3c15e3f27ef07bd78c040ae70c686293b22bebe84"}
Jul 17 22:30:26.514784 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:26.514703    2565 scope.go:117] "RemoveContainer" containerID="1982bc0efd497620ccb817d3c15e3f27ef07bd78c040ae70c686293b22bebe84"
Jul 17 22:30:26.654990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:26.654964    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq"
Jul 17 22:30:26.714847 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:26.714815    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc-maas-api-service-ca\") pod \"a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc\" (UID: \"a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc\") "
Jul 17 22:30:26.715255 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:26.715236    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc" (UID: "a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:30:26.816350 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:26.816308    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:30:27.520826 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:27.520791    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d51a5b0-fc3c-42f0-9ace-cf914878b659" containerID="e8405b5f00eb6a2de58360bcbe4d8f9c18ed193204f4689af91f38756d23ce5e" exitCode=6
Jul 17 22:30:27.521340 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:27.520879    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh" event={"ID":"5d51a5b0-fc3c-42f0-9ace-cf914878b659","Type":"ContainerDied","Data":"e8405b5f00eb6a2de58360bcbe4d8f9c18ed193204f4689af91f38756d23ce5e"}
Jul 17 22:30:27.521340 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:27.520946    2565 scope.go:117] "RemoveContainer" containerID="1982bc0efd497620ccb817d3c15e3f27ef07bd78c040ae70c686293b22bebe84"
Jul 17 22:30:27.521340 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:27.521137    2565 scope.go:117] "RemoveContainer" containerID="e8405b5f00eb6a2de58360bcbe4d8f9c18ed193204f4689af91f38756d23ce5e"
Jul 17 22:30:27.521495 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:30:27.521376    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh_odh-ai-gateway-infra(5d51a5b0-fc3c-42f0-9ace-cf914878b659)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh" podUID="5d51a5b0-fc3c-42f0-9ace-cf914878b659"
Jul 17 22:30:27.522226 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:27.522156    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq"
Jul 17 22:30:27.522226 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:27.522161    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq" event={"ID":"a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc","Type":"ContainerDied","Data":"c5af92231cf2dab42c33e0074640617671e9e762deb054762481f58693731533"}
Jul 17 22:30:27.532043 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:27.532025    2565 scope.go:117] "RemoveContainer" containerID="79a022b5d204df67b7bbcf5b3990794cd8dd152ffdf75ab063da86d37ec791c5"
Jul 17 22:30:27.555463 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:27.555437    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq"]
Jul 17 22:30:27.561128 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:27.561101    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-rntfq"]
Jul 17 22:30:28.528807 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:28.528776    2565 scope.go:117] "RemoveContainer" containerID="e8405b5f00eb6a2de58360bcbe4d8f9c18ed193204f4689af91f38756d23ce5e"
Jul 17 22:30:28.529267 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:30:28.529053    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh_odh-ai-gateway-infra(5d51a5b0-fc3c-42f0-9ace-cf914878b659)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh" podUID="5d51a5b0-fc3c-42f0-9ace-cf914878b659"
Jul 17 22:30:29.182518 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:29.182483    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc" path="/var/lib/kubelet/pods/a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc/volumes"
Jul 17 22:30:31.267634 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:31.267602    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:30:31.362857 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:31.362814    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:30:32.605773 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:32.605738    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:30:32.708970 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:32.708915    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:30:32.909206 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:32.909118    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:30:32.916134 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:32.916105    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-895e50b8-29738790-kfbvx"]
Jul 17 22:30:32.918930 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:32.918905    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-895e50b8-29738790-kfbvx"]
Jul 17 22:30:33.182553 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:33.182469    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cc61896-3906-412a-b6d6-ea91a856973d" path="/var/lib/kubelet/pods/9cc61896-3906-412a-b6d6-ea91a856973d/volumes"
Jul 17 22:30:36.846974 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:36.846938    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:30:37.136435 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:37.136339    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:30:37.297319 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:37.297277    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:30:37.309448 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:37.309418    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:30:42.176742 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.176713    2565 scope.go:117] "RemoveContainer" containerID="e8405b5f00eb6a2de58360bcbe4d8f9c18ed193204f4689af91f38756d23ce5e"
Jul 17 22:30:42.219299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.219269    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:30:42.602365 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.602332    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh" event={"ID":"5d51a5b0-fc3c-42f0-9ace-cf914878b659","Type":"ContainerStarted","Data":"cd20bdf750c04466de9daece2e53b2e83ae60bdfd4c83b452a749065d3454e53"}
Jul 17 22:30:42.640694 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.640641    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh" podStartSLOduration=18.640625993 podStartE2EDuration="18.640625993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:30:42.63614174 +0000 UTC m=+2706.025158390" watchObservedRunningTime="2026-07-17 22:30:42.640625993 +0000 UTC m=+2706.029642645"
Jul 17 22:30:42.766444 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.766412    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f"]
Jul 17 22:30:42.766945 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.766930    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc" containerName="revoke-keys"
Jul 17 22:30:42.767011 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.766950    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc" containerName="revoke-keys"
Jul 17 22:30:42.767011 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.766980    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc" containerName="revoke-keys"
Jul 17 22:30:42.767011 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.766989    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc" containerName="revoke-keys"
Jul 17 22:30:42.767011 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.767005    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc" containerName="revoke-keys"
Jul 17 22:30:42.767205 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.767013    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc" containerName="revoke-keys"
Jul 17 22:30:42.767205 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.767106    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc" containerName="revoke-keys"
Jul 17 22:30:42.767205 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.767121    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc" containerName="revoke-keys"
Jul 17 22:30:42.770597 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.770578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f"
Jul 17 22:30:42.786439 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.786412    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f"]
Jul 17 22:30:42.870522 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.870482    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a37b51-5402-4ab4-9bb2-6ace91a6f520-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-lws4f\" (UID: \"18a37b51-5402-4ab4-9bb2-6ace91a6f520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f"
Jul 17 22:30:42.972139 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.972093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a37b51-5402-4ab4-9bb2-6ace91a6f520-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-lws4f\" (UID: \"18a37b51-5402-4ab4-9bb2-6ace91a6f520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f"
Jul 17 22:30:42.972752 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:42.972732    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a37b51-5402-4ab4-9bb2-6ace91a6f520-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-lws4f\" (UID: \"18a37b51-5402-4ab4-9bb2-6ace91a6f520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f"
Jul 17 22:30:43.141419 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:43.141324    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f"
Jul 17 22:30:43.286309 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:43.286281    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f"]
Jul 17 22:30:43.287662 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:30:43.287624    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a37b51_5402_4ab4_9bb2_6ace91a6f520.slice/crio-cec207cd97038b14e3f18ebf389202d2e3c05786093bd99e2b9d32439fff67bf WatchSource:0}: Error finding container cec207cd97038b14e3f18ebf389202d2e3c05786093bd99e2b9d32439fff67bf: Status 404 returned error can't find the container with id cec207cd97038b14e3f18ebf389202d2e3c05786093bd99e2b9d32439fff67bf
Jul 17 22:30:43.609130 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:43.609096    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d51a5b0-fc3c-42f0-9ace-cf914878b659" containerID="cd20bdf750c04466de9daece2e53b2e83ae60bdfd4c83b452a749065d3454e53" exitCode=6
Jul 17 22:30:43.609328 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:43.609180    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh" event={"ID":"5d51a5b0-fc3c-42f0-9ace-cf914878b659","Type":"ContainerDied","Data":"cd20bdf750c04466de9daece2e53b2e83ae60bdfd4c83b452a749065d3454e53"}
Jul 17 22:30:43.609328 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:43.609256    2565 scope.go:117] "RemoveContainer" containerID="e8405b5f00eb6a2de58360bcbe4d8f9c18ed193204f4689af91f38756d23ce5e"
Jul 17 22:30:43.609655 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:43.609634    2565 scope.go:117] "RemoveContainer" containerID="cd20bdf750c04466de9daece2e53b2e83ae60bdfd4c83b452a749065d3454e53"
Jul 17 22:30:43.609892 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:30:43.609870    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh_odh-ai-gateway-infra(5d51a5b0-fc3c-42f0-9ace-cf914878b659)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh" podUID="5d51a5b0-fc3c-42f0-9ace-cf914878b659"
Jul 17 22:30:43.610797 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:43.610773    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f" event={"ID":"18a37b51-5402-4ab4-9bb2-6ace91a6f520","Type":"ContainerStarted","Data":"b5d867af3a19ef285c67831e5ceec4ff5e2327df6c4860f41b0ce091baea1dc0"}
Jul 17 22:30:43.610889 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:43.610817    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f" event={"ID":"18a37b51-5402-4ab4-9bb2-6ace91a6f520","Type":"ContainerStarted","Data":"cec207cd97038b14e3f18ebf389202d2e3c05786093bd99e2b9d32439fff67bf"}
Jul 17 22:30:43.651922 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:43.651884    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh"]
Jul 17 22:30:43.658810 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:43.658763    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f" podStartSLOduration=1.658749928 podStartE2EDuration="1.658749928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:30:43.656911923 +0000 UTC m=+2707.045928573" watchObservedRunningTime="2026-07-17 22:30:43.658749928 +0000 UTC m=+2707.047766579"
Jul 17 22:30:44.619728 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:44.619689    2565 generic.go:356] "Generic (PLEG): container finished" podID="18a37b51-5402-4ab4-9bb2-6ace91a6f520" containerID="b5d867af3a19ef285c67831e5ceec4ff5e2327df6c4860f41b0ce091baea1dc0" exitCode=6
Jul 17 22:30:44.620054 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:44.619727    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f" event={"ID":"18a37b51-5402-4ab4-9bb2-6ace91a6f520","Type":"ContainerDied","Data":"b5d867af3a19ef285c67831e5ceec4ff5e2327df6c4860f41b0ce091baea1dc0"}
Jul 17 22:30:44.620054 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:44.619974    2565 scope.go:117] "RemoveContainer" containerID="b5d867af3a19ef285c67831e5ceec4ff5e2327df6c4860f41b0ce091baea1dc0"
Jul 17 22:30:44.757362 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:44.757334    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh"
Jul 17 22:30:44.891445 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:44.891367    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d51a5b0-fc3c-42f0-9ace-cf914878b659-maas-api-service-ca\") pod \"5d51a5b0-fc3c-42f0-9ace-cf914878b659\" (UID: \"5d51a5b0-fc3c-42f0-9ace-cf914878b659\") "
Jul 17 22:30:44.891720 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:44.891698    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d51a5b0-fc3c-42f0-9ace-cf914878b659-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d51a5b0-fc3c-42f0-9ace-cf914878b659" (UID: "5d51a5b0-fc3c-42f0-9ace-cf914878b659"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:30:44.992797 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:44.992763    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d51a5b0-fc3c-42f0-9ace-cf914878b659-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:30:45.625944 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:45.625912    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh"
Jul 17 22:30:45.626430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:45.625912    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh" event={"ID":"5d51a5b0-fc3c-42f0-9ace-cf914878b659","Type":"ContainerDied","Data":"2c4f00d18e8e0773745d4b4f5e8134e7a2a7f94ecefe8fe7044e4fa08f887059"}
Jul 17 22:30:45.626430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:45.626061    2565 scope.go:117] "RemoveContainer" containerID="cd20bdf750c04466de9daece2e53b2e83ae60bdfd4c83b452a749065d3454e53"
Jul 17 22:30:45.628156 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:45.628125    2565 generic.go:356] "Generic (PLEG): container finished" podID="18a37b51-5402-4ab4-9bb2-6ace91a6f520" containerID="8888d10f77ac7e75053c6d555ce91d89360e29017262456258190541e8c614d2" exitCode=6
Jul 17 22:30:45.628267 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:45.628165    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f" event={"ID":"18a37b51-5402-4ab4-9bb2-6ace91a6f520","Type":"ContainerDied","Data":"8888d10f77ac7e75053c6d555ce91d89360e29017262456258190541e8c614d2"}
Jul 17 22:30:45.628675 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:45.628650    2565 scope.go:117] "RemoveContainer" containerID="8888d10f77ac7e75053c6d555ce91d89360e29017262456258190541e8c614d2"
Jul 17 22:30:45.628922 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:30:45.628900    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-lws4f_odh-ai-gateway-infra(18a37b51-5402-4ab4-9bb2-6ace91a6f520)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f" podUID="18a37b51-5402-4ab4-9bb2-6ace91a6f520"
Jul 17 22:30:45.634995 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:45.634976    2565 scope.go:117] "RemoveContainer" containerID="b5d867af3a19ef285c67831e5ceec4ff5e2327df6c4860f41b0ce091baea1dc0"
Jul 17 22:30:45.659725 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:45.659697    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh"]
Jul 17 22:30:45.664658 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:45.664629    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-rtcqh"]
Jul 17 22:30:46.635780 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:46.635745    2565 scope.go:117] "RemoveContainer" containerID="8888d10f77ac7e75053c6d555ce91d89360e29017262456258190541e8c614d2"
Jul 17 22:30:46.636196 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:30:46.635947    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-lws4f_odh-ai-gateway-infra(18a37b51-5402-4ab4-9bb2-6ace91a6f520)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f" podUID="18a37b51-5402-4ab4-9bb2-6ace91a6f520"
Jul 17 22:30:47.182622 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:30:47.182583    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d51a5b0-fc3c-42f0-9ace-cf914878b659" path="/var/lib/kubelet/pods/5d51a5b0-fc3c-42f0-9ace-cf914878b659/volumes"
Jul 17 22:31:00.177253 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:00.177220    2565 scope.go:117] "RemoveContainer" containerID="8888d10f77ac7e75053c6d555ce91d89360e29017262456258190541e8c614d2"
Jul 17 22:31:00.707502 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:00.707463    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f" event={"ID":"18a37b51-5402-4ab4-9bb2-6ace91a6f520","Type":"ContainerStarted","Data":"3a92273b7f4779ebed9e0dcfea878a54ee7b5316526cba4bf840e167bd19c786"}
Jul 17 22:31:01.713805 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:01.713766    2565 generic.go:356] "Generic (PLEG): container finished" podID="18a37b51-5402-4ab4-9bb2-6ace91a6f520" containerID="3a92273b7f4779ebed9e0dcfea878a54ee7b5316526cba4bf840e167bd19c786" exitCode=6
Jul 17 22:31:01.714298 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:01.713840    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f" event={"ID":"18a37b51-5402-4ab4-9bb2-6ace91a6f520","Type":"ContainerDied","Data":"3a92273b7f4779ebed9e0dcfea878a54ee7b5316526cba4bf840e167bd19c786"}
Jul 17 22:31:01.714298 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:01.713900    2565 scope.go:117] "RemoveContainer" containerID="8888d10f77ac7e75053c6d555ce91d89360e29017262456258190541e8c614d2"
Jul 17 22:31:01.714298 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:01.714200    2565 scope.go:117] "RemoveContainer" containerID="3a92273b7f4779ebed9e0dcfea878a54ee7b5316526cba4bf840e167bd19c786"
Jul 17 22:31:01.714439 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:31:01.714419    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-lws4f_odh-ai-gateway-infra(18a37b51-5402-4ab4-9bb2-6ace91a6f520)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f" podUID="18a37b51-5402-4ab4-9bb2-6ace91a6f520"
Jul 17 22:31:01.738323 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:01.738292    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f"]
Jul 17 22:31:02.858251 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:02.858226    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f"
Jul 17 22:31:02.866371 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:02.866345    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a37b51-5402-4ab4-9bb2-6ace91a6f520-maas-api-service-ca\") pod \"18a37b51-5402-4ab4-9bb2-6ace91a6f520\" (UID: \"18a37b51-5402-4ab4-9bb2-6ace91a6f520\") "
Jul 17 22:31:02.866653 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:02.866633    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a37b51-5402-4ab4-9bb2-6ace91a6f520-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18a37b51-5402-4ab4-9bb2-6ace91a6f520" (UID: "18a37b51-5402-4ab4-9bb2-6ace91a6f520"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:31:02.966945 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:02.966905    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a37b51-5402-4ab4-9bb2-6ace91a6f520-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:31:03.726171 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:03.726135    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f"
Jul 17 22:31:03.726352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:03.726141    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f" event={"ID":"18a37b51-5402-4ab4-9bb2-6ace91a6f520","Type":"ContainerDied","Data":"cec207cd97038b14e3f18ebf389202d2e3c05786093bd99e2b9d32439fff67bf"}
Jul 17 22:31:03.726352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:03.726273    2565 scope.go:117] "RemoveContainer" containerID="3a92273b7f4779ebed9e0dcfea878a54ee7b5316526cba4bf840e167bd19c786"
Jul 17 22:31:03.746145 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:03.746112    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f"]
Jul 17 22:31:03.754894 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:03.754863    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-lws4f"]
Jul 17 22:31:05.182396 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:05.182358    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18a37b51-5402-4ab4-9bb2-6ace91a6f520" path="/var/lib/kubelet/pods/18a37b51-5402-4ab4-9bb2-6ace91a6f520/volumes"
Jul 17 22:31:11.782306 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:11.782266    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:18.734448 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:18.734409    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:19.820769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:19.820730    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:25.594334 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:25.594299    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:25.696283 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:25.696245    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:29.442711 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:29.442677    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"]
Jul 17 22:31:29.443202 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:29.442936    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj" podUID="e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" containerName="istio-proxy" containerID="cri-o://467e66763f44e560f12ad0290bcd9f35d7f9af598502b73aa656a13df0aed12c" gracePeriod=30
Jul 17 22:31:29.475215 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:29.475180    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:29.682334 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:29.682296    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:30.718961 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:30.718898    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:30.808090 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:30.808035    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:30.908288 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:30.908245    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:31.611656 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:31.611627    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:31.710622 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:31.710593    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:31.808080 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:31.808040    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:33.157791 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:33.157758    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:33.647512 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:33.647477    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:31:34.702487 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.702461    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:31:34.880077 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.880044    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-podinfo\") pod \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") "
Jul 17 22:31:34.880277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.880109    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-data\") pod \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") "
Jul 17 22:31:34.880277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.880142    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns84l\" (UniqueName: \"kubernetes.io/projected/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-kube-api-access-ns84l\") pod \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") "
Jul 17 22:31:34.880277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.880170    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istiod-ca-cert\") pod \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") "
Jul 17 22:31:34.880277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.880222    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-workload-certs\") pod \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") "
Jul 17 22:31:34.880277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.880243    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-token\") pod \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") "
Jul 17 22:31:34.880577 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.880281    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-workload-socket\") pod \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") "
Jul 17 22:31:34.880577 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.880310    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-envoy\") pod \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") "
Jul 17 22:31:34.880577 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.880342    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-credential-socket\") pod \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\" (UID: \"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a\") "
Jul 17 22:31:34.880577 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.880529    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" (UID: "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:31:34.880790 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.880613    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" (UID: "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:31:34.880790 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.880705    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" (UID: "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:31:34.880960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.880814    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" (UID: "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:31:34.880960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.880823    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-credential-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:31:34.880960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.880842    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istiod-ca-cert\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:31:34.880960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.880857    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-workload-certs\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:31:34.881210 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.881192    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-data" (OuterVolumeSpecName: "istio-data") pod "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" (UID: "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:31:34.882874 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.882524    2565 generic.go:356] "Generic (PLEG): container finished" podID="e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" containerID="467e66763f44e560f12ad0290bcd9f35d7f9af598502b73aa656a13df0aed12c" exitCode=0
Jul 17 22:31:34.882874 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.882711    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj" event={"ID":"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a","Type":"ContainerDied","Data":"467e66763f44e560f12ad0290bcd9f35d7f9af598502b73aa656a13df0aed12c"}
Jul 17 22:31:34.882874 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.882767    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj" event={"ID":"e9ff01ba-e9bd-464c-9ca5-f715fc2b291a","Type":"ContainerDied","Data":"a649479fbd2419f76a30d7c552fa8179dd39cd2db6efbc34a585a9cd6ec491c5"}
Jul 17 22:31:34.882874 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.882798    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"
Jul 17 22:31:34.883235 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.882905    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" (UID: "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:31:34.883235 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.882802    2565 scope.go:117] "RemoveContainer" containerID="467e66763f44e560f12ad0290bcd9f35d7f9af598502b73aa656a13df0aed12c"
Jul 17 22:31:34.883235 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.883187    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-token" (OuterVolumeSpecName: "istio-token") pod "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" (UID: "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:31:34.883640 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.883615    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-kube-api-access-ns84l" (OuterVolumeSpecName: "kube-api-access-ns84l") pod "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" (UID: "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a"). InnerVolumeSpecName "kube-api-access-ns84l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:31:34.883911 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.883886    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" (UID: "e9ff01ba-e9bd-464c-9ca5-f715fc2b291a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:31:34.905371 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.905341    2565 scope.go:117] "RemoveContainer" containerID="c5a0b7076347523ed2ea40b433282a1fa14dcbbd208e1dbe65524f7d9693b9c7"
Jul 17 22:31:34.916189 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.916164    2565 scope.go:117] "RemoveContainer" containerID="467e66763f44e560f12ad0290bcd9f35d7f9af598502b73aa656a13df0aed12c"
Jul 17 22:31:34.916578 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:31:34.916554    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467e66763f44e560f12ad0290bcd9f35d7f9af598502b73aa656a13df0aed12c\": container with ID starting with 467e66763f44e560f12ad0290bcd9f35d7f9af598502b73aa656a13df0aed12c not found: ID does not exist" containerID="467e66763f44e560f12ad0290bcd9f35d7f9af598502b73aa656a13df0aed12c"
Jul 17 22:31:34.916620 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.916588    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467e66763f44e560f12ad0290bcd9f35d7f9af598502b73aa656a13df0aed12c"} err="failed to get container status \"467e66763f44e560f12ad0290bcd9f35d7f9af598502b73aa656a13df0aed12c\": rpc error: code = NotFound desc = could not find container \"467e66763f44e560f12ad0290bcd9f35d7f9af598502b73aa656a13df0aed12c\": container with ID starting with 467e66763f44e560f12ad0290bcd9f35d7f9af598502b73aa656a13df0aed12c not found: ID does not exist"
Jul 17 22:31:34.916620 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.916607    2565 scope.go:117] "RemoveContainer" containerID="c5a0b7076347523ed2ea40b433282a1fa14dcbbd208e1dbe65524f7d9693b9c7"
Jul 17 22:31:34.916927 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:31:34.916898    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a0b7076347523ed2ea40b433282a1fa14dcbbd208e1dbe65524f7d9693b9c7\": container with ID starting with c5a0b7076347523ed2ea40b433282a1fa14dcbbd208e1dbe65524f7d9693b9c7 not found: ID does not exist" containerID="c5a0b7076347523ed2ea40b433282a1fa14dcbbd208e1dbe65524f7d9693b9c7"
Jul 17 22:31:34.916978 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.916934    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a0b7076347523ed2ea40b433282a1fa14dcbbd208e1dbe65524f7d9693b9c7"} err="failed to get container status \"c5a0b7076347523ed2ea40b433282a1fa14dcbbd208e1dbe65524f7d9693b9c7\": rpc error: code = NotFound desc = could not find container \"c5a0b7076347523ed2ea40b433282a1fa14dcbbd208e1dbe65524f7d9693b9c7\": container with ID starting with c5a0b7076347523ed2ea40b433282a1fa14dcbbd208e1dbe65524f7d9693b9c7 not found: ID does not exist"
Jul 17 22:31:34.981633 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.981586    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns84l\" (UniqueName: \"kubernetes.io/projected/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-kube-api-access-ns84l\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:31:34.981633 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.981624    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-token\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:31:34.981633 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.981641    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-workload-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:31:34.981974 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.981649    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-envoy\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:31:34.981974 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.981657    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-podinfo\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:31:34.981974 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:34.981665    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a-istio-data\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:31:35.209446 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:35.209412    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"]
Jul 17 22:31:35.216376 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:35.216348    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-895e50b8-openshift-default-659bcf588c-p5nsj"]
Jul 17 22:31:37.182179 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:37.182150    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" path="/var/lib/kubelet/pods/e9ff01ba-e9bd-464c-9ca5-f715fc2b291a/volumes"
Jul 17 22:31:39.948911 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:39.948877    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:40.084950 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:40.084909    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:40.747409 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:40.747377    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:42.779170 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779123    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm"]
Jul 17 22:31:42.779645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779506    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a37b51-5402-4ab4-9bb2-6ace91a6f520" containerName="revoke-keys"
Jul 17 22:31:42.779645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779516    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a37b51-5402-4ab4-9bb2-6ace91a6f520" containerName="revoke-keys"
Jul 17 22:31:42.779645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779530    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a37b51-5402-4ab4-9bb2-6ace91a6f520" containerName="revoke-keys"
Jul 17 22:31:42.779645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779535    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a37b51-5402-4ab4-9bb2-6ace91a6f520" containerName="revoke-keys"
Jul 17 22:31:42.779645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779543    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a37b51-5402-4ab4-9bb2-6ace91a6f520" containerName="revoke-keys"
Jul 17 22:31:42.779645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779548    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a37b51-5402-4ab4-9bb2-6ace91a6f520" containerName="revoke-keys"
Jul 17 22:31:42.779645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779554    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d51a5b0-fc3c-42f0-9ace-cf914878b659" containerName="revoke-keys"
Jul 17 22:31:42.779645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779560    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d51a5b0-fc3c-42f0-9ace-cf914878b659" containerName="revoke-keys"
Jul 17 22:31:42.779645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779569    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" containerName="istio-proxy"
Jul 17 22:31:42.779645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779575    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" containerName="istio-proxy"
Jul 17 22:31:42.779645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779580    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" containerName="istio-proxy"
Jul 17 22:31:42.779645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779585    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" containerName="istio-proxy"
Jul 17 22:31:42.779645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779596    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d51a5b0-fc3c-42f0-9ace-cf914878b659" containerName="revoke-keys"
Jul 17 22:31:42.779645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779600    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d51a5b0-fc3c-42f0-9ace-cf914878b659" containerName="revoke-keys"
Jul 17 22:31:42.779645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779645    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" containerName="istio-proxy"
Jul 17 22:31:42.779645 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779652    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8809ed1-c9d7-4c50-828b-f5f8dadeb5fc" containerName="revoke-keys"
Jul 17 22:31:42.780163 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779660    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a37b51-5402-4ab4-9bb2-6ace91a6f520" containerName="revoke-keys"
Jul 17 22:31:42.780163 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779666    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d51a5b0-fc3c-42f0-9ace-cf914878b659" containerName="revoke-keys"
Jul 17 22:31:42.780163 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779672    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d51a5b0-fc3c-42f0-9ace-cf914878b659" containerName="revoke-keys"
Jul 17 22:31:42.780163 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779677    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a37b51-5402-4ab4-9bb2-6ace91a6f520" containerName="revoke-keys"
Jul 17 22:31:42.780163 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.779683    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d51a5b0-fc3c-42f0-9ace-cf914878b659" containerName="revoke-keys"
Jul 17 22:31:42.783772 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.783754    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm"
Jul 17 22:31:42.786693 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.786664    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:31:42.786824 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.786711    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:31:42.790687 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.790659    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm"]
Jul 17 22:31:42.850606 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.850569    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22074e72-c118-485d-b95b-ac0e2169aad9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm\" (UID: \"22074e72-c118-485d-b95b-ac0e2169aad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm"
Jul 17 22:31:42.952016 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.951975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22074e72-c118-485d-b95b-ac0e2169aad9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm\" (UID: \"22074e72-c118-485d-b95b-ac0e2169aad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm"
Jul 17 22:31:42.952618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:42.952599    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22074e72-c118-485d-b95b-ac0e2169aad9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm\" (UID: \"22074e72-c118-485d-b95b-ac0e2169aad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm"
Jul 17 22:31:43.096231 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:43.096139    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm"
Jul 17 22:31:43.247807 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:43.247773    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm"]
Jul 17 22:31:43.251447 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:31:43.251416    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22074e72_c118_485d_b95b_ac0e2169aad9.slice/crio-0c963983718c1cfe36bc0e78cf5e43d377e6345094d958dfe59289fd3fd70e5c WatchSource:0}: Error finding container 0c963983718c1cfe36bc0e78cf5e43d377e6345094d958dfe59289fd3fd70e5c: Status 404 returned error can't find the container with id 0c963983718c1cfe36bc0e78cf5e43d377e6345094d958dfe59289fd3fd70e5c
Jul 17 22:31:43.929682 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:43.929651    2565 generic.go:356] "Generic (PLEG): container finished" podID="22074e72-c118-485d-b95b-ac0e2169aad9" containerID="1f2e22ad9e555f8d0cd13a532e97ba649825ef68100de452577abeeda9d5c772" exitCode=6
Jul 17 22:31:43.930153 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:43.929741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm" event={"ID":"22074e72-c118-485d-b95b-ac0e2169aad9","Type":"ContainerDied","Data":"1f2e22ad9e555f8d0cd13a532e97ba649825ef68100de452577abeeda9d5c772"}
Jul 17 22:31:43.930153 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:43.929813    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm" event={"ID":"22074e72-c118-485d-b95b-ac0e2169aad9","Type":"ContainerStarted","Data":"0c963983718c1cfe36bc0e78cf5e43d377e6345094d958dfe59289fd3fd70e5c"}
Jul 17 22:31:43.930153 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:43.929901    2565 scope.go:117] "RemoveContainer" containerID="1f2e22ad9e555f8d0cd13a532e97ba649825ef68100de452577abeeda9d5c772"
Jul 17 22:31:44.937629 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:44.937531    2565 generic.go:356] "Generic (PLEG): container finished" podID="22074e72-c118-485d-b95b-ac0e2169aad9" containerID="e8c5e9d615cd1531df26e97755f6c06cadcd777f6671acde6cfd69ccd3da1c5f" exitCode=6
Jul 17 22:31:44.937629 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:44.937618    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm" event={"ID":"22074e72-c118-485d-b95b-ac0e2169aad9","Type":"ContainerDied","Data":"e8c5e9d615cd1531df26e97755f6c06cadcd777f6671acde6cfd69ccd3da1c5f"}
Jul 17 22:31:44.938101 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:44.937681    2565 scope.go:117] "RemoveContainer" containerID="1f2e22ad9e555f8d0cd13a532e97ba649825ef68100de452577abeeda9d5c772"
Jul 17 22:31:44.938101 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:44.937922    2565 scope.go:117] "RemoveContainer" containerID="e8c5e9d615cd1531df26e97755f6c06cadcd777f6671acde6cfd69ccd3da1c5f"
Jul 17 22:31:44.938200 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:31:44.938181    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm_odh-ai-gateway-infra(22074e72-c118-485d-b95b-ac0e2169aad9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm" podUID="22074e72-c118-485d-b95b-ac0e2169aad9"
Jul 17 22:31:45.945260 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:45.945231    2565 scope.go:117] "RemoveContainer" containerID="e8c5e9d615cd1531df26e97755f6c06cadcd777f6671acde6cfd69ccd3da1c5f"
Jul 17 22:31:45.945659 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:31:45.945432    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm_odh-ai-gateway-infra(22074e72-c118-485d-b95b-ac0e2169aad9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm" podUID="22074e72-c118-485d-b95b-ac0e2169aad9"
Jul 17 22:31:46.433592 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:46.433554    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl"]
Jul 17 22:31:46.433952 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:46.433939    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d51a5b0-fc3c-42f0-9ace-cf914878b659" containerName="revoke-keys"
Jul 17 22:31:46.434003 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:46.433955    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d51a5b0-fc3c-42f0-9ace-cf914878b659" containerName="revoke-keys"
Jul 17 22:31:46.434047 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:46.434038    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ff01ba-e9bd-464c-9ca5-f715fc2b291a" containerName="istio-proxy"
Jul 17 22:31:46.434113 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:46.434053    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a37b51-5402-4ab4-9bb2-6ace91a6f520" containerName="revoke-keys"
Jul 17 22:31:46.438334 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:46.438318    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl"
Jul 17 22:31:46.444628 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:46.444561    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl"]
Jul 17 22:31:46.486175 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:46.486133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a59680-960b-40c7-b3c1-fcfcc54fd0b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl\" (UID: \"22a59680-960b-40c7-b3c1-fcfcc54fd0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl"
Jul 17 22:31:46.587415 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:46.587367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a59680-960b-40c7-b3c1-fcfcc54fd0b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl\" (UID: \"22a59680-960b-40c7-b3c1-fcfcc54fd0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl"
Jul 17 22:31:46.588041 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:46.588019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a59680-960b-40c7-b3c1-fcfcc54fd0b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl\" (UID: \"22a59680-960b-40c7-b3c1-fcfcc54fd0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl"
Jul 17 22:31:46.750333 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:46.750297    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl"
Jul 17 22:31:46.762327 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:46.762296    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:46.873534 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:46.873501    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:46.903299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:46.903272    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl"]
Jul 17 22:31:46.904636 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:31:46.904611    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a59680_960b_40c7_b3c1_fcfcc54fd0b9.slice/crio-4b921f3b7c355a70ea2b8854bcfcc8242f6c0582c7c6c32a6ac168c937262239 WatchSource:0}: Error finding container 4b921f3b7c355a70ea2b8854bcfcc8242f6c0582c7c6c32a6ac168c937262239: Status 404 returned error can't find the container with id 4b921f3b7c355a70ea2b8854bcfcc8242f6c0582c7c6c32a6ac168c937262239
Jul 17 22:31:46.952051 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:46.952021    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl" event={"ID":"22a59680-960b-40c7-b3c1-fcfcc54fd0b9","Type":"ContainerStarted","Data":"4b921f3b7c355a70ea2b8854bcfcc8242f6c0582c7c6c32a6ac168c937262239"}
Jul 17 22:31:46.964514 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:46.964485    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:47.958415 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:47.958316    2565 generic.go:356] "Generic (PLEG): container finished" podID="22a59680-960b-40c7-b3c1-fcfcc54fd0b9" containerID="f210f33db5978a084b9c470918f94e8413bd7dbbf35fbf435fc2bc23b139aa02" exitCode=6
Jul 17 22:31:47.958415 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:47.958359    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl" event={"ID":"22a59680-960b-40c7-b3c1-fcfcc54fd0b9","Type":"ContainerDied","Data":"f210f33db5978a084b9c470918f94e8413bd7dbbf35fbf435fc2bc23b139aa02"}
Jul 17 22:31:47.958971 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:47.958625    2565 scope.go:117] "RemoveContainer" containerID="f210f33db5978a084b9c470918f94e8413bd7dbbf35fbf435fc2bc23b139aa02"
Jul 17 22:31:48.951878 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:48.951783    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:31:48.965477 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:48.965442    2565 generic.go:356] "Generic (PLEG): container finished" podID="22a59680-960b-40c7-b3c1-fcfcc54fd0b9" containerID="78162d255be00f2e3e01adebaec9920c34af69d09e2ef0383f464c08be913d15" exitCode=6
Jul 17 22:31:48.965973 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:48.965546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl" event={"ID":"22a59680-960b-40c7-b3c1-fcfcc54fd0b9","Type":"ContainerDied","Data":"78162d255be00f2e3e01adebaec9920c34af69d09e2ef0383f464c08be913d15"}
Jul 17 22:31:48.965973 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:48.965633    2565 scope.go:117] "RemoveContainer" containerID="f210f33db5978a084b9c470918f94e8413bd7dbbf35fbf435fc2bc23b139aa02"
Jul 17 22:31:48.965973 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:48.965747    2565 scope.go:117] "RemoveContainer" containerID="78162d255be00f2e3e01adebaec9920c34af69d09e2ef0383f464c08be913d15"
Jul 17 22:31:48.966192 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:31:48.966030    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl_odh-ai-gateway-infra(22a59680-960b-40c7-b3c1-fcfcc54fd0b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl" podUID="22a59680-960b-40c7-b3c1-fcfcc54fd0b9"
Jul 17 22:31:49.972588 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:49.972561    2565 scope.go:117] "RemoveContainer" containerID="78162d255be00f2e3e01adebaec9920c34af69d09e2ef0383f464c08be913d15"
Jul 17 22:31:49.973001 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:31:49.972753    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl_odh-ai-gateway-infra(22a59680-960b-40c7-b3c1-fcfcc54fd0b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl" podUID="22a59680-960b-40c7-b3c1-fcfcc54fd0b9"
Jul 17 22:31:57.513130 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:57.513054    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:57.841530 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:57.841440    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:31:58.197490 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:58.197405    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2"]
Jul 17 22:31:58.202151 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:58.202132    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2"
Jul 17 22:31:58.207704 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:58.207675    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2"]
Jul 17 22:31:58.300857 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:58.300820    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bafbcf6e-0ec2-4755-8557-8737f2dd5389-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2\" (UID: \"bafbcf6e-0ec2-4755-8557-8737f2dd5389\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2"
Jul 17 22:31:58.402460 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:58.402421    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bafbcf6e-0ec2-4755-8557-8737f2dd5389-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2\" (UID: \"bafbcf6e-0ec2-4755-8557-8737f2dd5389\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2"
Jul 17 22:31:58.403146 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:58.403121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bafbcf6e-0ec2-4755-8557-8737f2dd5389-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2\" (UID: \"bafbcf6e-0ec2-4755-8557-8737f2dd5389\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2"
Jul 17 22:31:58.514176 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:58.514139    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2"
Jul 17 22:31:58.655655 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:58.655631    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2"]
Jul 17 22:31:58.657962 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:31:58.657933    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafbcf6e_0ec2_4755_8557_8737f2dd5389.slice/crio-0fd2e8c553b2adc0dbc2c973b1df790315ac5d027b9c090678ec32d050c3ee5c WatchSource:0}: Error finding container 0fd2e8c553b2adc0dbc2c973b1df790315ac5d027b9c090678ec32d050c3ee5c: Status 404 returned error can't find the container with id 0fd2e8c553b2adc0dbc2c973b1df790315ac5d027b9c090678ec32d050c3ee5c
Jul 17 22:31:59.017147 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:59.017110    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2" event={"ID":"bafbcf6e-0ec2-4755-8557-8737f2dd5389","Type":"ContainerStarted","Data":"c5b07f796476320b08bf3e1e5d7757b7fe90294efc161ea7279d953e476917b4"}
Jul 17 22:31:59.017332 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:59.017172    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2" event={"ID":"bafbcf6e-0ec2-4755-8557-8737f2dd5389","Type":"ContainerStarted","Data":"0fd2e8c553b2adc0dbc2c973b1df790315ac5d027b9c090678ec32d050c3ee5c"}
Jul 17 22:31:59.033878 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:31:59.033752    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2" podStartSLOduration=1.03372977 podStartE2EDuration="1.03372977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:31:59.033123775 +0000 UTC m=+2782.422140427" watchObservedRunningTime="2026-07-17 22:31:59.03372977 +0000 UTC m=+2782.422746473"
Jul 17 22:32:00.023631 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:00.023601    2565 generic.go:356] "Generic (PLEG): container finished" podID="bafbcf6e-0ec2-4755-8557-8737f2dd5389" containerID="c5b07f796476320b08bf3e1e5d7757b7fe90294efc161ea7279d953e476917b4" exitCode=6
Jul 17 22:32:00.024037 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:00.023702    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2" event={"ID":"bafbcf6e-0ec2-4755-8557-8737f2dd5389","Type":"ContainerDied","Data":"c5b07f796476320b08bf3e1e5d7757b7fe90294efc161ea7279d953e476917b4"}
Jul 17 22:32:00.024037 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:00.023856    2565 scope.go:117] "RemoveContainer" containerID="c5b07f796476320b08bf3e1e5d7757b7fe90294efc161ea7279d953e476917b4"
Jul 17 22:32:01.030649 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:01.030610    2565 generic.go:356] "Generic (PLEG): container finished" podID="bafbcf6e-0ec2-4755-8557-8737f2dd5389" containerID="459227a2732e68f876eba6d67c733b27438a6792fc71f8f12e1182d8f48f6609" exitCode=6
Jul 17 22:32:01.031127 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:01.030697    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2" event={"ID":"bafbcf6e-0ec2-4755-8557-8737f2dd5389","Type":"ContainerDied","Data":"459227a2732e68f876eba6d67c733b27438a6792fc71f8f12e1182d8f48f6609"}
Jul 17 22:32:01.031127 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:01.030759    2565 scope.go:117] "RemoveContainer" containerID="c5b07f796476320b08bf3e1e5d7757b7fe90294efc161ea7279d953e476917b4"
Jul 17 22:32:01.031127 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:01.030950    2565 scope.go:117] "RemoveContainer" containerID="459227a2732e68f876eba6d67c733b27438a6792fc71f8f12e1182d8f48f6609"
Jul 17 22:32:01.031283 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:32:01.031191    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2_odh-ai-gateway-infra(bafbcf6e-0ec2-4755-8557-8737f2dd5389)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2" podUID="bafbcf6e-0ec2-4755-8557-8737f2dd5389"
Jul 17 22:32:01.177264 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:01.177231    2565 scope.go:117] "RemoveContainer" containerID="e8c5e9d615cd1531df26e97755f6c06cadcd777f6671acde6cfd69ccd3da1c5f"
Jul 17 22:32:02.038141 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:02.038038    2565 generic.go:356] "Generic (PLEG): container finished" podID="22074e72-c118-485d-b95b-ac0e2169aad9" containerID="30bc712cc89a84c8c10d3a2b1bef37e6722840efe9b536487e927ecc5fde25ca" exitCode=6
Jul 17 22:32:02.038141 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:02.038104    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm" event={"ID":"22074e72-c118-485d-b95b-ac0e2169aad9","Type":"ContainerDied","Data":"30bc712cc89a84c8c10d3a2b1bef37e6722840efe9b536487e927ecc5fde25ca"}
Jul 17 22:32:02.038643 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:02.038173    2565 scope.go:117] "RemoveContainer" containerID="e8c5e9d615cd1531df26e97755f6c06cadcd777f6671acde6cfd69ccd3da1c5f"
Jul 17 22:32:02.038643 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:02.038527    2565 scope.go:117] "RemoveContainer" containerID="30bc712cc89a84c8c10d3a2b1bef37e6722840efe9b536487e927ecc5fde25ca"
Jul 17 22:32:02.038773 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:32:02.038750    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm_odh-ai-gateway-infra(22074e72-c118-485d-b95b-ac0e2169aad9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm" podUID="22074e72-c118-485d-b95b-ac0e2169aad9"
Jul 17 22:32:02.040174 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:02.040158    2565 scope.go:117] "RemoveContainer" containerID="459227a2732e68f876eba6d67c733b27438a6792fc71f8f12e1182d8f48f6609"
Jul 17 22:32:02.040350 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:32:02.040332    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2_odh-ai-gateway-infra(bafbcf6e-0ec2-4755-8557-8737f2dd5389)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2" podUID="bafbcf6e-0ec2-4755-8557-8737f2dd5389"
Jul 17 22:32:03.065618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:03.065582    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm"]
Jul 17 22:32:03.212872 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:03.212839    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm"
Jul 17 22:32:03.350944 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:03.350847    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22074e72-c118-485d-b95b-ac0e2169aad9-maas-api-service-ca\") pod \"22074e72-c118-485d-b95b-ac0e2169aad9\" (UID: \"22074e72-c118-485d-b95b-ac0e2169aad9\") "
Jul 17 22:32:03.351294 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:03.351275    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22074e72-c118-485d-b95b-ac0e2169aad9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22074e72-c118-485d-b95b-ac0e2169aad9" (UID: "22074e72-c118-485d-b95b-ac0e2169aad9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:32:03.452450 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:03.452409    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22074e72-c118-485d-b95b-ac0e2169aad9-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:32:04.052055 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:04.052025    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm"
Jul 17 22:32:04.052055 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:04.052030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm" event={"ID":"22074e72-c118-485d-b95b-ac0e2169aad9","Type":"ContainerDied","Data":"0c963983718c1cfe36bc0e78cf5e43d377e6345094d958dfe59289fd3fd70e5c"}
Jul 17 22:32:04.052313 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:04.052115    2565 scope.go:117] "RemoveContainer" containerID="30bc712cc89a84c8c10d3a2b1bef37e6722840efe9b536487e927ecc5fde25ca"
Jul 17 22:32:04.073860 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:04.073827    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm"]
Jul 17 22:32:04.079799 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:04.079772    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hsvsm"]
Jul 17 22:32:04.176545 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:04.176505    2565 scope.go:117] "RemoveContainer" containerID="78162d255be00f2e3e01adebaec9920c34af69d09e2ef0383f464c08be913d15"
Jul 17 22:32:05.058983 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:05.058953    2565 generic.go:356] "Generic (PLEG): container finished" podID="22a59680-960b-40c7-b3c1-fcfcc54fd0b9" containerID="fb827ce13bfeba78171acbe83b07dfc048bcbfdc5ae600d995c9dd4e213c42ae" exitCode=6
Jul 17 22:32:05.059215 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:05.059031    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl" event={"ID":"22a59680-960b-40c7-b3c1-fcfcc54fd0b9","Type":"ContainerDied","Data":"fb827ce13bfeba78171acbe83b07dfc048bcbfdc5ae600d995c9dd4e213c42ae"}
Jul 17 22:32:05.059215 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:05.059120    2565 scope.go:117] "RemoveContainer" containerID="78162d255be00f2e3e01adebaec9920c34af69d09e2ef0383f464c08be913d15"
Jul 17 22:32:05.059435 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:05.059416    2565 scope.go:117] "RemoveContainer" containerID="fb827ce13bfeba78171acbe83b07dfc048bcbfdc5ae600d995c9dd4e213c42ae"
Jul 17 22:32:05.059693 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:32:05.059671    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl_odh-ai-gateway-infra(22a59680-960b-40c7-b3c1-fcfcc54fd0b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl" podUID="22a59680-960b-40c7-b3c1-fcfcc54fd0b9"
Jul 17 22:32:05.182819 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:05.182784    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22074e72-c118-485d-b95b-ac0e2169aad9" path="/var/lib/kubelet/pods/22074e72-c118-485d-b95b-ac0e2169aad9/volumes"
Jul 17 22:32:06.088814 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:06.088770    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl"]
Jul 17 22:32:06.234304 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:06.234275    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl"
Jul 17 22:32:06.379482 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:06.379378    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a59680-960b-40c7-b3c1-fcfcc54fd0b9-maas-api-service-ca\") pod \"22a59680-960b-40c7-b3c1-fcfcc54fd0b9\" (UID: \"22a59680-960b-40c7-b3c1-fcfcc54fd0b9\") "
Jul 17 22:32:06.379737 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:06.379714    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a59680-960b-40c7-b3c1-fcfcc54fd0b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22a59680-960b-40c7-b3c1-fcfcc54fd0b9" (UID: "22a59680-960b-40c7-b3c1-fcfcc54fd0b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:32:06.480857 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:06.480814    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a59680-960b-40c7-b3c1-fcfcc54fd0b9-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:32:07.072949 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:07.072911    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl" event={"ID":"22a59680-960b-40c7-b3c1-fcfcc54fd0b9","Type":"ContainerDied","Data":"4b921f3b7c355a70ea2b8854bcfcc8242f6c0582c7c6c32a6ac168c937262239"}
Jul 17 22:32:07.072949 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:07.072945    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl"
Jul 17 22:32:07.073253 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:07.072986    2565 scope.go:117] "RemoveContainer" containerID="fb827ce13bfeba78171acbe83b07dfc048bcbfdc5ae600d995c9dd4e213c42ae"
Jul 17 22:32:07.095745 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:07.095711    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl"]
Jul 17 22:32:07.109194 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:07.109168    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-wmmpl"]
Jul 17 22:32:07.184394 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:07.184360    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22a59680-960b-40c7-b3c1-fcfcc54fd0b9" path="/var/lib/kubelet/pods/22a59680-960b-40c7-b3c1-fcfcc54fd0b9/volumes"
Jul 17 22:32:14.924445 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924408    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9"]
Jul 17 22:32:14.924926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924767    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22074e72-c118-485d-b95b-ac0e2169aad9" containerName="revoke-keys"
Jul 17 22:32:14.924926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924777    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22074e72-c118-485d-b95b-ac0e2169aad9" containerName="revoke-keys"
Jul 17 22:32:14.924926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924786    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a59680-960b-40c7-b3c1-fcfcc54fd0b9" containerName="revoke-keys"
Jul 17 22:32:14.924926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924791    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a59680-960b-40c7-b3c1-fcfcc54fd0b9" containerName="revoke-keys"
Jul 17 22:32:14.924926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924799    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22074e72-c118-485d-b95b-ac0e2169aad9" containerName="revoke-keys"
Jul 17 22:32:14.924926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924804    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22074e72-c118-485d-b95b-ac0e2169aad9" containerName="revoke-keys"
Jul 17 22:32:14.924926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924812    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a59680-960b-40c7-b3c1-fcfcc54fd0b9" containerName="revoke-keys"
Jul 17 22:32:14.924926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924817    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a59680-960b-40c7-b3c1-fcfcc54fd0b9" containerName="revoke-keys"
Jul 17 22:32:14.924926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924832    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a59680-960b-40c7-b3c1-fcfcc54fd0b9" containerName="revoke-keys"
Jul 17 22:32:14.924926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924837    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a59680-960b-40c7-b3c1-fcfcc54fd0b9" containerName="revoke-keys"
Jul 17 22:32:14.924926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924885    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a59680-960b-40c7-b3c1-fcfcc54fd0b9" containerName="revoke-keys"
Jul 17 22:32:14.924926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924891    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a59680-960b-40c7-b3c1-fcfcc54fd0b9" containerName="revoke-keys"
Jul 17 22:32:14.924926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924898    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a59680-960b-40c7-b3c1-fcfcc54fd0b9" containerName="revoke-keys"
Jul 17 22:32:14.924926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924906    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="22074e72-c118-485d-b95b-ac0e2169aad9" containerName="revoke-keys"
Jul 17 22:32:14.924926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924912    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="22074e72-c118-485d-b95b-ac0e2169aad9" containerName="revoke-keys"
Jul 17 22:32:14.924926 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.924917    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="22074e72-c118-485d-b95b-ac0e2169aad9" containerName="revoke-keys"
Jul 17 22:32:14.927837 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.927820    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9"
Jul 17 22:32:14.950478 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:14.950445    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9"]
Jul 17 22:32:15.056715 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:15.056674    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8aaae1-f1aa-440a-90ab-442096f8521f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-trmc9\" (UID: \"8c8aaae1-f1aa-440a-90ab-442096f8521f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9"
Jul 17 22:32:15.158124 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:15.158042    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8aaae1-f1aa-440a-90ab-442096f8521f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-trmc9\" (UID: \"8c8aaae1-f1aa-440a-90ab-442096f8521f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9"
Jul 17 22:32:15.158715 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:15.158692    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8aaae1-f1aa-440a-90ab-442096f8521f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-trmc9\" (UID: \"8c8aaae1-f1aa-440a-90ab-442096f8521f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9"
Jul 17 22:32:15.238288 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:15.238180    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9"
Jul 17 22:32:15.400053 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:15.399945    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9"]
Jul 17 22:32:15.402549 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:32:15.402519    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8aaae1_f1aa_440a_90ab_442096f8521f.slice/crio-f57f56536a5c81b4dadc80d5a25415946e742292d12abc7ede76a1d83a8b23c4 WatchSource:0}: Error finding container f57f56536a5c81b4dadc80d5a25415946e742292d12abc7ede76a1d83a8b23c4: Status 404 returned error can't find the container with id f57f56536a5c81b4dadc80d5a25415946e742292d12abc7ede76a1d83a8b23c4
Jul 17 22:32:16.118997 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:16.118962    2565 generic.go:356] "Generic (PLEG): container finished" podID="8c8aaae1-f1aa-440a-90ab-442096f8521f" containerID="6f0a20e706b45e1e8ca92b6e47b5bdbe5517adb014a6adc264f0f2c3bed8b823" exitCode=6
Jul 17 22:32:16.119396 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:16.119010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9" event={"ID":"8c8aaae1-f1aa-440a-90ab-442096f8521f","Type":"ContainerDied","Data":"6f0a20e706b45e1e8ca92b6e47b5bdbe5517adb014a6adc264f0f2c3bed8b823"}
Jul 17 22:32:16.119396 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:16.119059    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9" event={"ID":"8c8aaae1-f1aa-440a-90ab-442096f8521f","Type":"ContainerStarted","Data":"f57f56536a5c81b4dadc80d5a25415946e742292d12abc7ede76a1d83a8b23c4"}
Jul 17 22:32:16.119396 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:16.119290    2565 scope.go:117] "RemoveContainer" containerID="6f0a20e706b45e1e8ca92b6e47b5bdbe5517adb014a6adc264f0f2c3bed8b823"
Jul 17 22:32:16.177575 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:16.177540    2565 scope.go:117] "RemoveContainer" containerID="459227a2732e68f876eba6d67c733b27438a6792fc71f8f12e1182d8f48f6609"
Jul 17 22:32:17.126680 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:17.126642    2565 generic.go:356] "Generic (PLEG): container finished" podID="bafbcf6e-0ec2-4755-8557-8737f2dd5389" containerID="5d00d2ec1097ea1148a16e08152fb758e1a6ce4f887d19f07ca17a24677978cb" exitCode=6
Jul 17 22:32:17.127166 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:17.126720    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2" event={"ID":"bafbcf6e-0ec2-4755-8557-8737f2dd5389","Type":"ContainerDied","Data":"5d00d2ec1097ea1148a16e08152fb758e1a6ce4f887d19f07ca17a24677978cb"}
Jul 17 22:32:17.127166 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:17.126783    2565 scope.go:117] "RemoveContainer" containerID="459227a2732e68f876eba6d67c733b27438a6792fc71f8f12e1182d8f48f6609"
Jul 17 22:32:17.127166 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:17.127139    2565 scope.go:117] "RemoveContainer" containerID="5d00d2ec1097ea1148a16e08152fb758e1a6ce4f887d19f07ca17a24677978cb"
Jul 17 22:32:17.127432 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:32:17.127400    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2_odh-ai-gateway-infra(bafbcf6e-0ec2-4755-8557-8737f2dd5389)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2" podUID="bafbcf6e-0ec2-4755-8557-8737f2dd5389"
Jul 17 22:32:17.128891 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:17.128873    2565 generic.go:356] "Generic (PLEG): container finished" podID="8c8aaae1-f1aa-440a-90ab-442096f8521f" containerID="557168327b851ebfbb8fd30ddfc48d73c49c8df1117bef657c0372ed3202c6f3" exitCode=6
Jul 17 22:32:17.128982 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:17.128905    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9" event={"ID":"8c8aaae1-f1aa-440a-90ab-442096f8521f","Type":"ContainerDied","Data":"557168327b851ebfbb8fd30ddfc48d73c49c8df1117bef657c0372ed3202c6f3"}
Jul 17 22:32:17.129146 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:17.129131    2565 scope.go:117] "RemoveContainer" containerID="557168327b851ebfbb8fd30ddfc48d73c49c8df1117bef657c0372ed3202c6f3"
Jul 17 22:32:17.129355 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:32:17.129339    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-trmc9_odh-ai-gateway-infra(8c8aaae1-f1aa-440a-90ab-442096f8521f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9" podUID="8c8aaae1-f1aa-440a-90ab-442096f8521f"
Jul 17 22:32:17.138328 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:17.138310    2565 scope.go:117] "RemoveContainer" containerID="6f0a20e706b45e1e8ca92b6e47b5bdbe5517adb014a6adc264f0f2c3bed8b823"
Jul 17 22:32:18.140304 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:18.140276    2565 scope.go:117] "RemoveContainer" containerID="557168327b851ebfbb8fd30ddfc48d73c49c8df1117bef657c0372ed3202c6f3"
Jul 17 22:32:18.140695 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:32:18.140458    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-trmc9_odh-ai-gateway-infra(8c8aaae1-f1aa-440a-90ab-442096f8521f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9" podUID="8c8aaae1-f1aa-440a-90ab-442096f8521f"
Jul 17 22:32:18.188512 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:18.188474    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2"]
Jul 17 22:32:18.335376 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:18.335349    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2"
Jul 17 22:32:18.489403 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:18.489309    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bafbcf6e-0ec2-4755-8557-8737f2dd5389-maas-api-service-ca\") pod \"bafbcf6e-0ec2-4755-8557-8737f2dd5389\" (UID: \"bafbcf6e-0ec2-4755-8557-8737f2dd5389\") "
Jul 17 22:32:18.489720 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:18.489699    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bafbcf6e-0ec2-4755-8557-8737f2dd5389-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bafbcf6e-0ec2-4755-8557-8737f2dd5389" (UID: "bafbcf6e-0ec2-4755-8557-8737f2dd5389"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:32:18.590311 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:18.590276    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bafbcf6e-0ec2-4755-8557-8737f2dd5389-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:32:18.889421 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:18.889382    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:19.146286 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:19.146200    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2"
Jul 17 22:32:19.146286 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:19.146199    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2" event={"ID":"bafbcf6e-0ec2-4755-8557-8737f2dd5389","Type":"ContainerDied","Data":"0fd2e8c553b2adc0dbc2c973b1df790315ac5d027b9c090678ec32d050c3ee5c"}
Jul 17 22:32:19.146777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:19.146338    2565 scope.go:117] "RemoveContainer" containerID="5d00d2ec1097ea1148a16e08152fb758e1a6ce4f887d19f07ca17a24677978cb"
Jul 17 22:32:19.182798 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:19.182771    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2"]
Jul 17 22:32:19.182798 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:19.182802    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-xt6n2"]
Jul 17 22:32:21.181821 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:21.181778    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bafbcf6e-0ec2-4755-8557-8737f2dd5389" path="/var/lib/kubelet/pods/bafbcf6e-0ec2-4755-8557-8737f2dd5389/volumes"
Jul 17 22:32:26.461596 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:26.461561    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:26.847194 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:26.847161    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:27.044158 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.044122    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:27.326490 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.326453    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"]
Jul 17 22:32:27.326878 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.326864    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22074e72-c118-485d-b95b-ac0e2169aad9" containerName="revoke-keys"
Jul 17 22:32:27.326923 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.326881    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22074e72-c118-485d-b95b-ac0e2169aad9" containerName="revoke-keys"
Jul 17 22:32:27.326923 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.326892    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bafbcf6e-0ec2-4755-8557-8737f2dd5389" containerName="revoke-keys"
Jul 17 22:32:27.326923 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.326897    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bafbcf6e-0ec2-4755-8557-8737f2dd5389" containerName="revoke-keys"
Jul 17 22:32:27.326923 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.326906    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bafbcf6e-0ec2-4755-8557-8737f2dd5389" containerName="revoke-keys"
Jul 17 22:32:27.326923 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.326915    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bafbcf6e-0ec2-4755-8557-8737f2dd5389" containerName="revoke-keys"
Jul 17 22:32:27.327101 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.326980    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bafbcf6e-0ec2-4755-8557-8737f2dd5389" containerName="revoke-keys"
Jul 17 22:32:27.327101 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.326992    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bafbcf6e-0ec2-4755-8557-8737f2dd5389" containerName="revoke-keys"
Jul 17 22:32:27.331395 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.331374    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.335330 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.335304    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-11aecf-openshift-default-dockercfg-x7w8p\""
Jul 17 22:32:27.368829 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.368795    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"]
Jul 17 22:32:27.446705 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.446672    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:27.469024 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.468982    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-workload-socket\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.469024 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.469024    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istiod-ca-cert\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.469553 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.469096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-workload-certs\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.469553 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.469121    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-podinfo\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.469553 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.469190    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-credential-socket\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.469553 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.469256    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-data\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.469553 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.469291    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-envoy\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.469553 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.469312    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-token\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.469553 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.469342    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt5hp\" (UniqueName: \"kubernetes.io/projected/2ba2521a-b4f2-48c9-a99c-ec688f51d605-kube-api-access-nt5hp\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.569922 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.569882    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt5hp\" (UniqueName: \"kubernetes.io/projected/2ba2521a-b4f2-48c9-a99c-ec688f51d605-kube-api-access-nt5hp\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.570190 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.569960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-workload-socket\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.570190 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.569985    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istiod-ca-cert\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.570190 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.570031    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-workload-certs\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.570190 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.570056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-podinfo\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.570190 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.570166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-credential-socket\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.570480 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.570213    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-data\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.570480 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.570242    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-envoy\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.570480 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.570265    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-token\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.570630 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.570495    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-workload-certs\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.570630 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.570506    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-workload-socket\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.570630 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.570588    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-data\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.570825 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.570641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-credential-socket\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.570963 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.570947    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istiod-ca-cert\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.572685 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.572659    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-envoy\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.572785 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.572768    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-podinfo\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.595011 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.594932    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt5hp\" (UniqueName: \"kubernetes.io/projected/2ba2521a-b4f2-48c9-a99c-ec688f51d605-kube-api-access-nt5hp\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.600341 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.600320    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-token\") pod \"e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") " pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.615738 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.615709    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:27.641748 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.641712    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:27.814940 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.814873    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"]
Jul 17 22:32:27.817715 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:32:27.817683    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba2521a_b4f2_48c9_a99c_ec688f51d605.slice/crio-42efccb74db1c0dc13e8ee7f98b63fdb91ab567c6f090aef70c618171fbdcd5a WatchSource:0}: Error finding container 42efccb74db1c0dc13e8ee7f98b63fdb91ab567c6f090aef70c618171fbdcd5a: Status 404 returned error can't find the container with id 42efccb74db1c0dc13e8ee7f98b63fdb91ab567c6f090aef70c618171fbdcd5a
Jul 17 22:32:27.819793 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.819759    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:32:27.819883 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.819829    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:32:27.819883 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:27.819854    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:32:28.191611 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:28.191506    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn" event={"ID":"2ba2521a-b4f2-48c9-a99c-ec688f51d605","Type":"ContainerStarted","Data":"9e9a23191cd98047911448d2874871380a31dbb400f0e08cdc2805ce5b5af9ea"}
Jul 17 22:32:28.191611 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:28.191592    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn" event={"ID":"2ba2521a-b4f2-48c9-a99c-ec688f51d605","Type":"ContainerStarted","Data":"42efccb74db1c0dc13e8ee7f98b63fdb91ab567c6f090aef70c618171fbdcd5a"}
Jul 17 22:32:28.242740 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:28.242686    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn" podStartSLOduration=1.242670881 podStartE2EDuration="1.242670881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:32:28.242190862 +0000 UTC m=+2811.631207531" watchObservedRunningTime="2026-07-17 22:32:28.242670881 +0000 UTC m=+2811.631687532"
Jul 17 22:32:28.642543 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:28.642502    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:28.647562 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:28.647538    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:29.197008 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:29.196970    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:29.198169 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:29.198148    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:30.486795 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:30.486757    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:31.182915 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:31.182883    2565 scope.go:117] "RemoveContainer" containerID="557168327b851ebfbb8fd30ddfc48d73c49c8df1117bef657c0372ed3202c6f3"
Jul 17 22:32:32.194851 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:32.194754    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:32.212983 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:32.212952    2565 generic.go:356] "Generic (PLEG): container finished" podID="8c8aaae1-f1aa-440a-90ab-442096f8521f" containerID="dea6a2410122f4a90896663ef00a14fb809d29dad95556e3d3196af04677ec47" exitCode=6
Jul 17 22:32:32.213181 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:32.213024    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9" event={"ID":"8c8aaae1-f1aa-440a-90ab-442096f8521f","Type":"ContainerDied","Data":"dea6a2410122f4a90896663ef00a14fb809d29dad95556e3d3196af04677ec47"}
Jul 17 22:32:32.213181 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:32.213111    2565 scope.go:117] "RemoveContainer" containerID="557168327b851ebfbb8fd30ddfc48d73c49c8df1117bef657c0372ed3202c6f3"
Jul 17 22:32:32.213410 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:32.213392    2565 scope.go:117] "RemoveContainer" containerID="dea6a2410122f4a90896663ef00a14fb809d29dad95556e3d3196af04677ec47"
Jul 17 22:32:32.213681 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:32:32.213646    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-trmc9_odh-ai-gateway-infra(8c8aaae1-f1aa-440a-90ab-442096f8521f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9" podUID="8c8aaae1-f1aa-440a-90ab-442096f8521f"
Jul 17 22:32:33.289961 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:33.289921    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9"]
Jul 17 22:32:33.427995 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:33.427967    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9"
Jul 17 22:32:33.513497 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:33.513462    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8aaae1-f1aa-440a-90ab-442096f8521f-maas-api-service-ca\") pod \"8c8aaae1-f1aa-440a-90ab-442096f8521f\" (UID: \"8c8aaae1-f1aa-440a-90ab-442096f8521f\") "
Jul 17 22:32:33.513862 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:33.513820    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8aaae1-f1aa-440a-90ab-442096f8521f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c8aaae1-f1aa-440a-90ab-442096f8521f" (UID: "8c8aaae1-f1aa-440a-90ab-442096f8521f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:32:33.614491 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:33.614405    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8aaae1-f1aa-440a-90ab-442096f8521f-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:32:34.225816 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:34.225780    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9" event={"ID":"8c8aaae1-f1aa-440a-90ab-442096f8521f","Type":"ContainerDied","Data":"f57f56536a5c81b4dadc80d5a25415946e742292d12abc7ede76a1d83a8b23c4"}
Jul 17 22:32:34.226002 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:34.225797    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9"
Jul 17 22:32:34.226002 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:34.225857    2565 scope.go:117] "RemoveContainer" containerID="dea6a2410122f4a90896663ef00a14fb809d29dad95556e3d3196af04677ec47"
Jul 17 22:32:34.270778 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:34.270741    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9"]
Jul 17 22:32:34.278359 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:34.278323    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-trmc9"]
Jul 17 22:32:35.008126 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.008060    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:35.183147 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.183117    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c8aaae1-f1aa-440a-90ab-442096f8521f" path="/var/lib/kubelet/pods/8c8aaae1-f1aa-440a-90ab-442096f8521f/volumes"
Jul 17 22:32:35.338495 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.338399    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:35.728280 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.728189    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"]
Jul 17 22:32:35.728557 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.728542    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c8aaae1-f1aa-440a-90ab-442096f8521f" containerName="revoke-keys"
Jul 17 22:32:35.728617 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.728559    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c8aaae1-f1aa-440a-90ab-442096f8521f" containerName="revoke-keys"
Jul 17 22:32:35.728617 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.728579    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bafbcf6e-0ec2-4755-8557-8737f2dd5389" containerName="revoke-keys"
Jul 17 22:32:35.728617 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.728585    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bafbcf6e-0ec2-4755-8557-8737f2dd5389" containerName="revoke-keys"
Jul 17 22:32:35.728617 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.728594    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c8aaae1-f1aa-440a-90ab-442096f8521f" containerName="revoke-keys"
Jul 17 22:32:35.728617 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.728599    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c8aaae1-f1aa-440a-90ab-442096f8521f" containerName="revoke-keys"
Jul 17 22:32:35.728868 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.728652    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c8aaae1-f1aa-440a-90ab-442096f8521f" containerName="revoke-keys"
Jul 17 22:32:35.728868 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.728659    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c8aaae1-f1aa-440a-90ab-442096f8521f" containerName="revoke-keys"
Jul 17 22:32:35.728868 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.728665    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bafbcf6e-0ec2-4755-8557-8737f2dd5389" containerName="revoke-keys"
Jul 17 22:32:35.728868 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.728671    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c8aaae1-f1aa-440a-90ab-442096f8521f" containerName="revoke-keys"
Jul 17 22:32:35.732893 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.732872    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.738930 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.738908    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4ccee5-openshift-default-dockercfg-jpxzk\""
Jul 17 22:32:35.776448 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.776410    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"]
Jul 17 22:32:35.835168 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.835133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-workload-certs\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.835350 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.835237    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-credential-socket\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.835350 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.835272    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-token\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.835350 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.835308    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-workload-socket\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.835350 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.835337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-podinfo\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.835491 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.835402    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istiod-ca-cert\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.835491 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.835463    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-envoy\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.835491 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.835485    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxb7s\" (UniqueName: \"kubernetes.io/projected/6ebc939d-7c34-4aa1-be7b-90801d6f536c-kube-api-access-vxb7s\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.835586 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.835513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-data\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.856761 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.856717    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:35.936536 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.936492    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-credential-socket\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.936536 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.936542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-token\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.936781 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.936570    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-workload-socket\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.936781 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.936595    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-podinfo\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.936781 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.936624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istiod-ca-cert\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.936781 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.936655    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-envoy\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.936781 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.936683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxb7s\" (UniqueName: \"kubernetes.io/projected/6ebc939d-7c34-4aa1-be7b-90801d6f536c-kube-api-access-vxb7s\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.936781 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.936715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-data\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.936781 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.936745    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-workload-certs\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.937194 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.937018    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-credential-socket\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.937194 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.937045    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-workload-socket\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.937194 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.937178    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-workload-certs\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.937306 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.937270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-data\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.937464 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.937445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istiod-ca-cert\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.939631 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.939604    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-envoy\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.939790 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.939768    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-podinfo\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.953720 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.953683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-token\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:35.957265 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:35.957234    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxb7s\" (UniqueName: \"kubernetes.io/projected/6ebc939d-7c34-4aa1-be7b-90801d6f536c-kube-api-access-vxb7s\") pod \"e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") " pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:36.001666 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:36.001628    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:36.043038 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:36.042995    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:36.217675 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:36.217648    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"]
Jul 17 22:32:36.220367 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:32:36.220342    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebc939d_7c34_4aa1_be7b_90801d6f536c.slice/crio-f3787647cd8eb266330c57dad36d9f24bf9cc39c3e1bc60181d4b9715656e006 WatchSource:0}: Error finding container f3787647cd8eb266330c57dad36d9f24bf9cc39c3e1bc60181d4b9715656e006: Status 404 returned error can't find the container with id f3787647cd8eb266330c57dad36d9f24bf9cc39c3e1bc60181d4b9715656e006
Jul 17 22:32:36.222624 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:36.222590    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:32:36.222691 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:36.222661    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:32:36.222723 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:36.222689    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:32:36.238258 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:36.238220    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs" event={"ID":"6ebc939d-7c34-4aa1-be7b-90801d6f536c","Type":"ContainerStarted","Data":"f3787647cd8eb266330c57dad36d9f24bf9cc39c3e1bc60181d4b9715656e006"}
Jul 17 22:32:37.244546 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:37.244506    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs" event={"ID":"6ebc939d-7c34-4aa1-be7b-90801d6f536c","Type":"ContainerStarted","Data":"e0e62752dd23754746afd4c0a2ea6500449cf91b7f108b89b01b25503c193237"}
Jul 17 22:32:37.283490 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:37.283435    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs" podStartSLOduration=2.2834177110000002 podStartE2EDuration="2.283417711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:32:37.281475238 +0000 UTC m=+2820.670491888" watchObservedRunningTime="2026-07-17 22:32:37.283417711 +0000 UTC m=+2820.672434361"
Jul 17 22:32:38.044223 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:38.044183    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:38.049560 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:38.049529    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:38.249834 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:38.249800    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:38.250728 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:38.250711    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:38.788688 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:38.788652    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:32:39.402817 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:39.402786    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:39.570402 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:39.570367    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:39.723974 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:39.723873    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:40.016319 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:40.016279    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:40.713767 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:40.713738    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:40.932178 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:40.932140    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:43.995281 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:43.995250    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:45.238305 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:45.238252    2565 patch_prober.go:28] interesting pod/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 17 22:32:45.238832 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:45.238313    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn" podUID="2ba2521a-b4f2-48c9-a99c-ec688f51d605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 17 22:32:45.636228 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:45.636186    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:46.532960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:46.532920    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst"]
Jul 17 22:32:46.533571 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:46.533481    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c8aaae1-f1aa-440a-90ab-442096f8521f" containerName="revoke-keys"
Jul 17 22:32:46.533571 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:46.533501    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c8aaae1-f1aa-440a-90ab-442096f8521f" containerName="revoke-keys"
Jul 17 22:32:46.536796 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:46.536775    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst"
Jul 17 22:32:46.547244 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:46.547216    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:32:46.547409 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:46.547252    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:32:46.575804 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:46.575771    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst"]
Jul 17 22:32:46.641843 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:46.641801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d795396f-5103-4a63-a90c-003eb661a615-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-gbbst\" (UID: \"d795396f-5103-4a63-a90c-003eb661a615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst"
Jul 17 22:32:46.742730 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:46.742672    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d795396f-5103-4a63-a90c-003eb661a615-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-gbbst\" (UID: \"d795396f-5103-4a63-a90c-003eb661a615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst"
Jul 17 22:32:46.743752 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:46.743702    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d795396f-5103-4a63-a90c-003eb661a615-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-gbbst\" (UID: \"d795396f-5103-4a63-a90c-003eb661a615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst"
Jul 17 22:32:46.847419 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:46.847333    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst"
Jul 17 22:32:47.092611 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:47.092580    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst"]
Jul 17 22:32:47.106174 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:32:47.106147    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd795396f_5103_4a63_a90c_003eb661a615.slice/crio-6f92154df2ead4f70509f702fdbfc947f947b50093db9f57b4beb072f0c49161 WatchSource:0}: Error finding container 6f92154df2ead4f70509f702fdbfc947f947b50093db9f57b4beb072f0c49161: Status 404 returned error can't find the container with id 6f92154df2ead4f70509f702fdbfc947f947b50093db9f57b4beb072f0c49161
Jul 17 22:32:47.306367 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:47.306327    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst" event={"ID":"d795396f-5103-4a63-a90c-003eb661a615","Type":"ContainerStarted","Data":"6f92154df2ead4f70509f702fdbfc947f947b50093db9f57b4beb072f0c49161"}
Jul 17 22:32:47.307745 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:47.307722    2565 generic.go:356] "Generic (PLEG): container finished" podID="2ba2521a-b4f2-48c9-a99c-ec688f51d605" containerID="9e9a23191cd98047911448d2874871380a31dbb400f0e08cdc2805ce5b5af9ea" exitCode=0
Jul 17 22:32:47.307843 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:47.307793    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn" event={"ID":"2ba2521a-b4f2-48c9-a99c-ec688f51d605","Type":"ContainerDied","Data":"9e9a23191cd98047911448d2874871380a31dbb400f0e08cdc2805ce5b5af9ea"}
Jul 17 22:32:47.308221 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:47.308205    2565 scope.go:117] "RemoveContainer" containerID="9e9a23191cd98047911448d2874871380a31dbb400f0e08cdc2805ce5b5af9ea"
Jul 17 22:32:47.309314 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:47.309281    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:32:47.309402 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:47.309358    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:32:47.309441 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:47.309397    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:32:48.316630 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:48.316538    2565 generic.go:356] "Generic (PLEG): container finished" podID="d795396f-5103-4a63-a90c-003eb661a615" containerID="b195dc96a621b46af38c9595676b24febc953bfb6c12207430b13ae723c4237a" exitCode=6
Jul 17 22:32:48.317014 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:48.316629    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst" event={"ID":"d795396f-5103-4a63-a90c-003eb661a615","Type":"ContainerDied","Data":"b195dc96a621b46af38c9595676b24febc953bfb6c12207430b13ae723c4237a"}
Jul 17 22:32:48.317014 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:48.316820    2565 scope.go:117] "RemoveContainer" containerID="b195dc96a621b46af38c9595676b24febc953bfb6c12207430b13ae723c4237a"
Jul 17 22:32:48.318965 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:48.318940    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn" event={"ID":"2ba2521a-b4f2-48c9-a99c-ec688f51d605","Type":"ContainerStarted","Data":"98580891d6bb4d83713e076e1659c57e4644bf76575e75d8016ee47982e910ea"}
Jul 17 22:32:48.642136 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:48.642098    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:48.642518 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:48.642493    2565 patch_prober.go:28] interesting pod/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Jul 17 22:32:48.642586 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:48.642545    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn" podUID="2ba2521a-b4f2-48c9-a99c-ec688f51d605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Jul 17 22:32:49.326789 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:49.326355    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst" event={"ID":"d795396f-5103-4a63-a90c-003eb661a615","Type":"ContainerStarted","Data":"0e1dcce56ee2d1881a03f084ecfdf82f316b96010eb2af4e0531e97f0e2f4e4a"}
Jul 17 22:32:49.365271 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:49.365188    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst" podStartSLOduration=3.365164697 podStartE2EDuration="3.365164697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:32:49.360057257 +0000 UTC m=+2832.749073908" watchObservedRunningTime="2026-07-17 22:32:49.365164697 +0000 UTC m=+2832.754181343"
Jul 17 22:32:49.642543 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:49.642435    2565 patch_prober.go:28] interesting pod/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Jul 17 22:32:49.642543 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:49.642505    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn" podUID="2ba2521a-b4f2-48c9-a99c-ec688f51d605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Jul 17 22:32:50.332908 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:50.332876    2565 generic.go:356] "Generic (PLEG): container finished" podID="d795396f-5103-4a63-a90c-003eb661a615" containerID="0e1dcce56ee2d1881a03f084ecfdf82f316b96010eb2af4e0531e97f0e2f4e4a" exitCode=6
Jul 17 22:32:50.333320 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:50.333001    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst" event={"ID":"d795396f-5103-4a63-a90c-003eb661a615","Type":"ContainerDied","Data":"0e1dcce56ee2d1881a03f084ecfdf82f316b96010eb2af4e0531e97f0e2f4e4a"}
Jul 17 22:32:50.333320 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:50.333041    2565 scope.go:117] "RemoveContainer" containerID="b195dc96a621b46af38c9595676b24febc953bfb6c12207430b13ae723c4237a"
Jul 17 22:32:50.333320 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:50.333208    2565 scope.go:117] "RemoveContainer" containerID="0e1dcce56ee2d1881a03f084ecfdf82f316b96010eb2af4e0531e97f0e2f4e4a"
Jul 17 22:32:50.333423 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:32:50.333410    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-gbbst_odh-ai-gateway-infra(d795396f-5103-4a63-a90c-003eb661a615)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst" podUID="d795396f-5103-4a63-a90c-003eb661a615"
Jul 17 22:32:50.642975 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:50.642878    2565 patch_prober.go:28] interesting pod/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Jul 17 22:32:50.642975 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:50.642937    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn" podUID="2ba2521a-b4f2-48c9-a99c-ec688f51d605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Jul 17 22:32:51.340696 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:51.340624    2565 scope.go:117] "RemoveContainer" containerID="0e1dcce56ee2d1881a03f084ecfdf82f316b96010eb2af4e0531e97f0e2f4e4a"
Jul 17 22:32:51.341055 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:32:51.340820    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-gbbst_odh-ai-gateway-infra(d795396f-5103-4a63-a90c-003eb661a615)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst" podUID="d795396f-5103-4a63-a90c-003eb661a615"
Jul 17 22:32:51.389332 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:51.389291    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:32:51.642751 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:51.642664    2565 patch_prober.go:28] interesting pod/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Jul 17 22:32:51.642901 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:51.642740    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn" podUID="2ba2521a-b4f2-48c9-a99c-ec688f51d605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Jul 17 22:32:52.646119 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:52.646083    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:52.646616 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:52.646356    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:52.647156 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:52.647136    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:32:53.086235 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:53.086199    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:53.332266 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:53.332229    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:32:54.115509 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:54.114193    2565 patch_prober.go:28] interesting pod/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": read tcp 10.133.0.2:42736->10.133.0.110:15021: read: connection reset by peer" start-of-body=
Jul 17 22:32:54.115509 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:54.114283    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs" podUID="6ebc939d-7c34-4aa1-be7b-90801d6f536c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": read tcp 10.133.0.2:42736->10.133.0.110:15021: read: connection reset by peer"
Jul 17 22:32:54.357865 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:54.357776    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ebc939d-7c34-4aa1-be7b-90801d6f536c" containerID="e0e62752dd23754746afd4c0a2ea6500449cf91b7f108b89b01b25503c193237" exitCode=0
Jul 17 22:32:54.357865 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:54.357846    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs" event={"ID":"6ebc939d-7c34-4aa1-be7b-90801d6f536c","Type":"ContainerDied","Data":"e0e62752dd23754746afd4c0a2ea6500449cf91b7f108b89b01b25503c193237"}
Jul 17 22:32:54.358479 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:54.358461    2565 scope.go:117] "RemoveContainer" containerID="e0e62752dd23754746afd4c0a2ea6500449cf91b7f108b89b01b25503c193237"
Jul 17 22:32:54.359547 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:54.359516    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:32:54.359662 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:54.359600    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:32:54.359662 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:54.359650    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:32:55.367773 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:55.367727    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs" event={"ID":"6ebc939d-7c34-4aa1-be7b-90801d6f536c","Type":"ContainerStarted","Data":"f1d0420b1a45d9a7e09c3442b9fe016aef5432acb09739797ad12ab1a317ec55"}
Jul 17 22:32:56.043341 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:56.043295    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:32:56.043585 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:56.043563    2565 patch_prober.go:28] interesting pod/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Jul 17 22:32:56.043636 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:56.043613    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs" podUID="6ebc939d-7c34-4aa1-be7b-90801d6f536c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Jul 17 22:32:57.043602 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:57.043564    2565 patch_prober.go:28] interesting pod/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Jul 17 22:32:57.044002 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:57.043622    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs" podUID="6ebc939d-7c34-4aa1-be7b-90801d6f536c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Jul 17 22:32:58.043990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:58.043947    2565 patch_prober.go:28] interesting pod/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Jul 17 22:32:58.044436 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:58.044011    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs" podUID="6ebc939d-7c34-4aa1-be7b-90801d6f536c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Jul 17 22:32:59.043898 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:59.043862    2565 patch_prober.go:28] interesting pod/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Jul 17 22:32:59.044102 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:32:59.043917    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs" podUID="6ebc939d-7c34-4aa1-be7b-90801d6f536c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Jul 17 22:33:00.047425 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:00.047394    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:33:00.047818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:00.047643    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:33:00.048422 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:00.048407    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:33:01.679562 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:01.679532    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:33:03.719896 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:03.719859    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:33:03.911707 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:03.911671    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:33:04.176957 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:04.176928    2565 scope.go:117] "RemoveContainer" containerID="0e1dcce56ee2d1881a03f084ecfdf82f316b96010eb2af4e0531e97f0e2f4e4a"
Jul 17 22:33:05.429030 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:05.428997    2565 generic.go:356] "Generic (PLEG): container finished" podID="d795396f-5103-4a63-a90c-003eb661a615" containerID="9ab472aa07c295c21f79aa28177b15c37112bb7889188e3d190f3d80345908c5" exitCode=6
Jul 17 22:33:05.429560 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:05.429088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst" event={"ID":"d795396f-5103-4a63-a90c-003eb661a615","Type":"ContainerDied","Data":"9ab472aa07c295c21f79aa28177b15c37112bb7889188e3d190f3d80345908c5"}
Jul 17 22:33:05.429560 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:05.429157    2565 scope.go:117] "RemoveContainer" containerID="0e1dcce56ee2d1881a03f084ecfdf82f316b96010eb2af4e0531e97f0e2f4e4a"
Jul 17 22:33:05.429560 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:05.429464    2565 scope.go:117] "RemoveContainer" containerID="9ab472aa07c295c21f79aa28177b15c37112bb7889188e3d190f3d80345908c5"
Jul 17 22:33:05.429727 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:33:05.429700    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-gbbst_odh-ai-gateway-infra(d795396f-5103-4a63-a90c-003eb661a615)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst" podUID="d795396f-5103-4a63-a90c-003eb661a615"
Jul 17 22:33:06.458039 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:06.457994    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst"]
Jul 17 22:33:06.602791 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:06.602763    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst"
Jul 17 22:33:06.756854 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:06.756822    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d795396f-5103-4a63-a90c-003eb661a615-maas-api-service-ca\") pod \"d795396f-5103-4a63-a90c-003eb661a615\" (UID: \"d795396f-5103-4a63-a90c-003eb661a615\") "
Jul 17 22:33:06.757318 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:06.757291    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d795396f-5103-4a63-a90c-003eb661a615-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d795396f-5103-4a63-a90c-003eb661a615" (UID: "d795396f-5103-4a63-a90c-003eb661a615"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:33:06.858510 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:06.858472    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d795396f-5103-4a63-a90c-003eb661a615-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:33:07.443612 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:07.443521    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst"
Jul 17 22:33:07.443612 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:07.443537    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst" event={"ID":"d795396f-5103-4a63-a90c-003eb661a615","Type":"ContainerDied","Data":"6f92154df2ead4f70509f702fdbfc947f947b50093db9f57b4beb072f0c49161"}
Jul 17 22:33:07.443612 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:07.443605    2565 scope.go:117] "RemoveContainer" containerID="9ab472aa07c295c21f79aa28177b15c37112bb7889188e3d190f3d80345908c5"
Jul 17 22:33:07.468204 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:07.466942    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst"]
Jul 17 22:33:07.474672 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:07.474646    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-gbbst"]
Jul 17 22:33:09.182723 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:09.182684    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d795396f-5103-4a63-a90c-003eb661a615" path="/var/lib/kubelet/pods/d795396f-5103-4a63-a90c-003eb661a615/volumes"
Jul 17 22:33:22.164789 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:22.164750    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:33:26.553058 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:26.553014    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:33:40.551443 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:40.551403    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:33:42.542030 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:42.542000    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:33:43.437690 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:43.437647    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:33:43.551441 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:43.551399    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs"]
Jul 17 22:33:43.551953 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:43.551933    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d795396f-5103-4a63-a90c-003eb661a615" containerName="revoke-keys"
Jul 17 22:33:43.552038 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:43.551956    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d795396f-5103-4a63-a90c-003eb661a615" containerName="revoke-keys"
Jul 17 22:33:43.552038 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:43.551971    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d795396f-5103-4a63-a90c-003eb661a615" containerName="revoke-keys"
Jul 17 22:33:43.552038 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:43.551978    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d795396f-5103-4a63-a90c-003eb661a615" containerName="revoke-keys"
Jul 17 22:33:43.552236 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:43.552120    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d795396f-5103-4a63-a90c-003eb661a615" containerName="revoke-keys"
Jul 17 22:33:43.552236 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:43.552137    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d795396f-5103-4a63-a90c-003eb661a615" containerName="revoke-keys"
Jul 17 22:33:43.556722 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:43.556700    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs"
Jul 17 22:33:43.560463 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:43.560435    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:33:43.560757 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:43.560734    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:33:43.563697 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:43.563671    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs"]
Jul 17 22:33:43.699931 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:43.699843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2efb5d-9490-43af-ba4d-9e5cde98129e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs\" (UID: \"4e2efb5d-9490-43af-ba4d-9e5cde98129e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs"
Jul 17 22:33:43.800595 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:43.800549    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2efb5d-9490-43af-ba4d-9e5cde98129e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs\" (UID: \"4e2efb5d-9490-43af-ba4d-9e5cde98129e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs"
Jul 17 22:33:43.801295 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:43.801272    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2efb5d-9490-43af-ba4d-9e5cde98129e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs\" (UID: \"4e2efb5d-9490-43af-ba4d-9e5cde98129e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs"
Jul 17 22:33:43.869236 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:43.869202    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs"
Jul 17 22:33:44.011203 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:44.011178    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs"]
Jul 17 22:33:44.013762 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:33:44.013723    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2efb5d_9490_43af_ba4d_9e5cde98129e.slice/crio-699580657acd41367553ba87d86ebbf44578aa391f2ad3d4922212384d399435 WatchSource:0}: Error finding container 699580657acd41367553ba87d86ebbf44578aa391f2ad3d4922212384d399435: Status 404 returned error can't find the container with id 699580657acd41367553ba87d86ebbf44578aa391f2ad3d4922212384d399435
Jul 17 22:33:44.091540 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:44.091507    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:33:44.196183 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:44.196147    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:33:44.635003 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:44.634916    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs" event={"ID":"4e2efb5d-9490-43af-ba4d-9e5cde98129e","Type":"ContainerStarted","Data":"05e765fb333aabacb124d5cbb868ca0be25852156674559b2bc6302fadd028ac"}
Jul 17 22:33:44.635003 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:44.634975    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs" event={"ID":"4e2efb5d-9490-43af-ba4d-9e5cde98129e","Type":"ContainerStarted","Data":"699580657acd41367553ba87d86ebbf44578aa391f2ad3d4922212384d399435"}
Jul 17 22:33:44.653001 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:44.652945    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs" podStartSLOduration=1.652928494 podStartE2EDuration="1.652928494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:33:44.650887946 +0000 UTC m=+2888.039904597" watchObservedRunningTime="2026-07-17 22:33:44.652928494 +0000 UTC m=+2888.041945144"
Jul 17 22:33:45.641711 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:45.641673    2565 generic.go:356] "Generic (PLEG): container finished" podID="4e2efb5d-9490-43af-ba4d-9e5cde98129e" containerID="05e765fb333aabacb124d5cbb868ca0be25852156674559b2bc6302fadd028ac" exitCode=0
Jul 17 22:33:45.642194 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:45.641738    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs" event={"ID":"4e2efb5d-9490-43af-ba4d-9e5cde98129e","Type":"ContainerDied","Data":"05e765fb333aabacb124d5cbb868ca0be25852156674559b2bc6302fadd028ac"}
Jul 17 22:33:46.553881 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:46.553843    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d"]
Jul 17 22:33:46.554261 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:46.554247    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d795396f-5103-4a63-a90c-003eb661a615" containerName="revoke-keys"
Jul 17 22:33:46.554320 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:46.554263    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d795396f-5103-4a63-a90c-003eb661a615" containerName="revoke-keys"
Jul 17 22:33:46.554320 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:46.554318    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d795396f-5103-4a63-a90c-003eb661a615" containerName="revoke-keys"
Jul 17 22:33:46.557334 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:46.557316    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d"
Jul 17 22:33:46.565825 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:46.565799    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d"]
Jul 17 22:33:46.629366 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:46.629330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac788942-8d12-4484-86f6-9861e2b6711f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-tls5d\" (UID: \"ac788942-8d12-4484-86f6-9861e2b6711f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d"
Jul 17 22:33:46.730510 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:46.730471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac788942-8d12-4484-86f6-9861e2b6711f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-tls5d\" (UID: \"ac788942-8d12-4484-86f6-9861e2b6711f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d"
Jul 17 22:33:46.731330 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:46.731305    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac788942-8d12-4484-86f6-9861e2b6711f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-tls5d\" (UID: \"ac788942-8d12-4484-86f6-9861e2b6711f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d"
Jul 17 22:33:46.784849 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:46.784821    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs"
Jul 17 22:33:46.869241 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:46.869158    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d"
Jul 17 22:33:46.932780 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:46.932382    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2efb5d-9490-43af-ba4d-9e5cde98129e-maas-api-service-ca\") pod \"4e2efb5d-9490-43af-ba4d-9e5cde98129e\" (UID: \"4e2efb5d-9490-43af-ba4d-9e5cde98129e\") "
Jul 17 22:33:46.933290 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:46.933142    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2efb5d-9490-43af-ba4d-9e5cde98129e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e2efb5d-9490-43af-ba4d-9e5cde98129e" (UID: "4e2efb5d-9490-43af-ba4d-9e5cde98129e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:33:47.016682 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:47.016657    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d"]
Jul 17 22:33:47.018622 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:33:47.018596    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac788942_8d12_4484_86f6_9861e2b6711f.slice/crio-ae92fa04d3a7d9bc1c254dde62f0aed9f9d0836150417dce1d05cec49974ef73 WatchSource:0}: Error finding container ae92fa04d3a7d9bc1c254dde62f0aed9f9d0836150417dce1d05cec49974ef73: Status 404 returned error can't find the container with id ae92fa04d3a7d9bc1c254dde62f0aed9f9d0836150417dce1d05cec49974ef73
Jul 17 22:33:47.033221 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:47.033196    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2efb5d-9490-43af-ba4d-9e5cde98129e-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:33:47.653307 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:47.653275    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs"
Jul 17 22:33:47.653503 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:47.653274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs" event={"ID":"4e2efb5d-9490-43af-ba4d-9e5cde98129e","Type":"ContainerDied","Data":"699580657acd41367553ba87d86ebbf44578aa391f2ad3d4922212384d399435"}
Jul 17 22:33:47.653503 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:47.653417    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="699580657acd41367553ba87d86ebbf44578aa391f2ad3d4922212384d399435"
Jul 17 22:33:47.654975 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:47.654942    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d" event={"ID":"ac788942-8d12-4484-86f6-9861e2b6711f","Type":"ContainerStarted","Data":"9aab580ffaea3a42e225429fd7a24bead8a4efc7dd69761ab69b05288809ac80"}
Jul 17 22:33:47.655155 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:47.654999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d" event={"ID":"ac788942-8d12-4484-86f6-9861e2b6711f","Type":"ContainerStarted","Data":"ae92fa04d3a7d9bc1c254dde62f0aed9f9d0836150417dce1d05cec49974ef73"}
Jul 17 22:33:47.672934 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:47.672883    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d" podStartSLOduration=1.6728673619999999 podStartE2EDuration="1.672867362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:33:47.670774098 +0000 UTC m=+2891.059790773" watchObservedRunningTime="2026-07-17 22:33:47.672867362 +0000 UTC m=+2891.061883992"
Jul 17 22:33:48.661196 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:48.661158    2565 generic.go:356] "Generic (PLEG): container finished" podID="ac788942-8d12-4484-86f6-9861e2b6711f" containerID="9aab580ffaea3a42e225429fd7a24bead8a4efc7dd69761ab69b05288809ac80" exitCode=6
Jul 17 22:33:48.661620 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:48.661244    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d" event={"ID":"ac788942-8d12-4484-86f6-9861e2b6711f","Type":"ContainerDied","Data":"9aab580ffaea3a42e225429fd7a24bead8a4efc7dd69761ab69b05288809ac80"}
Jul 17 22:33:48.661620 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:48.661417    2565 scope.go:117] "RemoveContainer" containerID="9aab580ffaea3a42e225429fd7a24bead8a4efc7dd69761ab69b05288809ac80"
Jul 17 22:33:49.667701 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:49.667609    2565 generic.go:356] "Generic (PLEG): container finished" podID="ac788942-8d12-4484-86f6-9861e2b6711f" containerID="514c41cff6c241e2701fbc9ec4829a67c714e5c21bd60544bd3dc792b08c7618" exitCode=6
Jul 17 22:33:49.668186 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:49.667693    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d" event={"ID":"ac788942-8d12-4484-86f6-9861e2b6711f","Type":"ContainerDied","Data":"514c41cff6c241e2701fbc9ec4829a67c714e5c21bd60544bd3dc792b08c7618"}
Jul 17 22:33:49.668186 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:49.667753    2565 scope.go:117] "RemoveContainer" containerID="9aab580ffaea3a42e225429fd7a24bead8a4efc7dd69761ab69b05288809ac80"
Jul 17 22:33:49.668186 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:49.667942    2565 scope.go:117] "RemoveContainer" containerID="514c41cff6c241e2701fbc9ec4829a67c714e5c21bd60544bd3dc792b08c7618"
Jul 17 22:33:49.668332 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:33:49.668209    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-tls5d_odh-ai-gateway-infra(ac788942-8d12-4484-86f6-9861e2b6711f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d" podUID="ac788942-8d12-4484-86f6-9861e2b6711f"
Jul 17 22:33:50.674752 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:50.674724    2565 scope.go:117] "RemoveContainer" containerID="514c41cff6c241e2701fbc9ec4829a67c714e5c21bd60544bd3dc792b08c7618"
Jul 17 22:33:50.675164 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:33:50.674929    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-tls5d_odh-ai-gateway-infra(ac788942-8d12-4484-86f6-9861e2b6711f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d" podUID="ac788942-8d12-4484-86f6-9861e2b6711f"
Jul 17 22:33:53.932179 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:53.932149    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:33:54.062035 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:54.062001    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:33:56.157194 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:56.157154    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:33:58.652278 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:33:58.652241    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:01.176779 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:01.176743    2565 scope.go:117] "RemoveContainer" containerID="514c41cff6c241e2701fbc9ec4829a67c714e5c21bd60544bd3dc792b08c7618"
Jul 17 22:34:01.729777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:01.729743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d" event={"ID":"ac788942-8d12-4484-86f6-9861e2b6711f","Type":"ContainerStarted","Data":"66c3f83d99820392f6d1a2079c54a0185483ab27207c7c855178c9bd2b8a21c1"}
Jul 17 22:34:02.736984 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:02.736949    2565 generic.go:356] "Generic (PLEG): container finished" podID="ac788942-8d12-4484-86f6-9861e2b6711f" containerID="66c3f83d99820392f6d1a2079c54a0185483ab27207c7c855178c9bd2b8a21c1" exitCode=6
Jul 17 22:34:02.737432 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:02.737018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d" event={"ID":"ac788942-8d12-4484-86f6-9861e2b6711f","Type":"ContainerDied","Data":"66c3f83d99820392f6d1a2079c54a0185483ab27207c7c855178c9bd2b8a21c1"}
Jul 17 22:34:02.737432 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:02.737093    2565 scope.go:117] "RemoveContainer" containerID="514c41cff6c241e2701fbc9ec4829a67c714e5c21bd60544bd3dc792b08c7618"
Jul 17 22:34:02.737432 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:02.737415    2565 scope.go:117] "RemoveContainer" containerID="66c3f83d99820392f6d1a2079c54a0185483ab27207c7c855178c9bd2b8a21c1"
Jul 17 22:34:02.737661 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:34:02.737638    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-tls5d_odh-ai-gateway-infra(ac788942-8d12-4484-86f6-9861e2b6711f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d" podUID="ac788942-8d12-4484-86f6-9861e2b6711f"
Jul 17 22:34:02.763680 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:02.763640    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d"]
Jul 17 22:34:03.884356 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:03.884326    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d"
Jul 17 22:34:03.892364 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:03.892314    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac788942-8d12-4484-86f6-9861e2b6711f-maas-api-service-ca\") pod \"ac788942-8d12-4484-86f6-9861e2b6711f\" (UID: \"ac788942-8d12-4484-86f6-9861e2b6711f\") "
Jul 17 22:34:03.892608 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:03.892587    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac788942-8d12-4484-86f6-9861e2b6711f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac788942-8d12-4484-86f6-9861e2b6711f" (UID: "ac788942-8d12-4484-86f6-9861e2b6711f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:34:03.993575 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:03.993544    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac788942-8d12-4484-86f6-9861e2b6711f-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:04.749860 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:04.749830    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d"
Jul 17 22:34:04.750056 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:04.749865    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d" event={"ID":"ac788942-8d12-4484-86f6-9861e2b6711f","Type":"ContainerDied","Data":"ae92fa04d3a7d9bc1c254dde62f0aed9f9d0836150417dce1d05cec49974ef73"}
Jul 17 22:34:04.750056 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:04.749932    2565 scope.go:117] "RemoveContainer" containerID="66c3f83d99820392f6d1a2079c54a0185483ab27207c7c855178c9bd2b8a21c1"
Jul 17 22:34:04.772472 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:04.772425    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d"]
Jul 17 22:34:04.774916 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:04.774894    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tls5d"]
Jul 17 22:34:04.825591 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:04.825558    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:04.926016 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:04.925988    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:05.182983 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:05.182899    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac788942-8d12-4484-86f6-9861e2b6711f" path="/var/lib/kubelet/pods/ac788942-8d12-4484-86f6-9861e2b6711f/volumes"
Jul 17 22:34:09.319255 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:09.319219    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"]
Jul 17 22:34:09.319895 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:09.319473    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn" podUID="2ba2521a-b4f2-48c9-a99c-ec688f51d605" containerName="istio-proxy" containerID="cri-o://98580891d6bb4d83713e076e1659c57e4644bf76575e75d8016ee47982e910ea" gracePeriod=30
Jul 17 22:34:09.341685 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:09.341650    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:09.581561 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:09.581480    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:10.356808 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:10.356767    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:10.458211 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:10.458177    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:14.582498 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.582428    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:34:14.696633 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.696596    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-workload-certs\") pod \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") "
Jul 17 22:34:14.696633 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.696633    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-envoy\") pod \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") "
Jul 17 22:34:14.696843 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.696669    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-podinfo\") pod \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") "
Jul 17 22:34:14.696843 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.696726    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istiod-ca-cert\") pod \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") "
Jul 17 22:34:14.696843 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.696757    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-credential-socket\") pod \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") "
Jul 17 22:34:14.696843 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.696796    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt5hp\" (UniqueName: \"kubernetes.io/projected/2ba2521a-b4f2-48c9-a99c-ec688f51d605-kube-api-access-nt5hp\") pod \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") "
Jul 17 22:34:14.696843 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.696823    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-data\") pod \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") "
Jul 17 22:34:14.697126 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.696864    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-token\") pod \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") "
Jul 17 22:34:14.697126 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.696928    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-workload-socket\") pod \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\" (UID: \"2ba2521a-b4f2-48c9-a99c-ec688f51d605\") "
Jul 17 22:34:14.697126 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.696935    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2ba2521a-b4f2-48c9-a99c-ec688f51d605" (UID: "2ba2521a-b4f2-48c9-a99c-ec688f51d605"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:34:14.697299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.697200    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2ba2521a-b4f2-48c9-a99c-ec688f51d605" (UID: "2ba2521a-b4f2-48c9-a99c-ec688f51d605"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:34:14.697299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.697257    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2ba2521a-b4f2-48c9-a99c-ec688f51d605" (UID: "2ba2521a-b4f2-48c9-a99c-ec688f51d605"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:34:14.697405 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.697301    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-credential-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:14.697405 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.697321    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-workload-certs\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:14.697405 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.697394    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2ba2521a-b4f2-48c9-a99c-ec688f51d605" (UID: "2ba2521a-b4f2-48c9-a99c-ec688f51d605"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:34:14.697851 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.697821    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-data" (OuterVolumeSpecName: "istio-data") pod "2ba2521a-b4f2-48c9-a99c-ec688f51d605" (UID: "2ba2521a-b4f2-48c9-a99c-ec688f51d605"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:34:14.699152 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.699121    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2ba2521a-b4f2-48c9-a99c-ec688f51d605" (UID: "2ba2521a-b4f2-48c9-a99c-ec688f51d605"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:34:14.699239 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.699221    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-token" (OuterVolumeSpecName: "istio-token") pod "2ba2521a-b4f2-48c9-a99c-ec688f51d605" (UID: "2ba2521a-b4f2-48c9-a99c-ec688f51d605"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:34:14.699492 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.699478    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2ba2521a-b4f2-48c9-a99c-ec688f51d605" (UID: "2ba2521a-b4f2-48c9-a99c-ec688f51d605"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:34:14.699641 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.699617    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba2521a-b4f2-48c9-a99c-ec688f51d605-kube-api-access-nt5hp" (OuterVolumeSpecName: "kube-api-access-nt5hp") pod "2ba2521a-b4f2-48c9-a99c-ec688f51d605" (UID: "2ba2521a-b4f2-48c9-a99c-ec688f51d605"). InnerVolumeSpecName "kube-api-access-nt5hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:34:14.798256 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.798218    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-envoy\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:14.798256 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.798249    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-podinfo\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:14.798256 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.798261    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istiod-ca-cert\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:14.798256 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.798269    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt5hp\" (UniqueName: \"kubernetes.io/projected/2ba2521a-b4f2-48c9-a99c-ec688f51d605-kube-api-access-nt5hp\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:14.798539 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.798279    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-data\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:14.798539 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.798287    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba2521a-b4f2-48c9-a99c-ec688f51d605-istio-token\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:14.798539 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.798295    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba2521a-b4f2-48c9-a99c-ec688f51d605-workload-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:14.808677 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.808639    2565 generic.go:356] "Generic (PLEG): container finished" podID="2ba2521a-b4f2-48c9-a99c-ec688f51d605" containerID="98580891d6bb4d83713e076e1659c57e4644bf76575e75d8016ee47982e910ea" exitCode=0
Jul 17 22:34:14.808870 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.808712    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"
Jul 17 22:34:14.808870 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.808734    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn" event={"ID":"2ba2521a-b4f2-48c9-a99c-ec688f51d605","Type":"ContainerDied","Data":"98580891d6bb4d83713e076e1659c57e4644bf76575e75d8016ee47982e910ea"}
Jul 17 22:34:14.808870 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.808826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn" event={"ID":"2ba2521a-b4f2-48c9-a99c-ec688f51d605","Type":"ContainerDied","Data":"42efccb74db1c0dc13e8ee7f98b63fdb91ab567c6f090aef70c618171fbdcd5a"}
Jul 17 22:34:14.808870 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.808855    2565 scope.go:117] "RemoveContainer" containerID="98580891d6bb4d83713e076e1659c57e4644bf76575e75d8016ee47982e910ea"
Jul 17 22:34:14.818283 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.818258    2565 scope.go:117] "RemoveContainer" containerID="9e9a23191cd98047911448d2874871380a31dbb400f0e08cdc2805ce5b5af9ea"
Jul 17 22:34:14.826595 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.826574    2565 scope.go:117] "RemoveContainer" containerID="98580891d6bb4d83713e076e1659c57e4644bf76575e75d8016ee47982e910ea"
Jul 17 22:34:14.826881 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:34:14.826862    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98580891d6bb4d83713e076e1659c57e4644bf76575e75d8016ee47982e910ea\": container with ID starting with 98580891d6bb4d83713e076e1659c57e4644bf76575e75d8016ee47982e910ea not found: ID does not exist" containerID="98580891d6bb4d83713e076e1659c57e4644bf76575e75d8016ee47982e910ea"
Jul 17 22:34:14.826941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.826891    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98580891d6bb4d83713e076e1659c57e4644bf76575e75d8016ee47982e910ea"} err="failed to get container status \"98580891d6bb4d83713e076e1659c57e4644bf76575e75d8016ee47982e910ea\": rpc error: code = NotFound desc = could not find container \"98580891d6bb4d83713e076e1659c57e4644bf76575e75d8016ee47982e910ea\": container with ID starting with 98580891d6bb4d83713e076e1659c57e4644bf76575e75d8016ee47982e910ea not found: ID does not exist"
Jul 17 22:34:14.826941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.826909    2565 scope.go:117] "RemoveContainer" containerID="9e9a23191cd98047911448d2874871380a31dbb400f0e08cdc2805ce5b5af9ea"
Jul 17 22:34:14.827165 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:34:14.827148    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e9a23191cd98047911448d2874871380a31dbb400f0e08cdc2805ce5b5af9ea\": container with ID starting with 9e9a23191cd98047911448d2874871380a31dbb400f0e08cdc2805ce5b5af9ea not found: ID does not exist" containerID="9e9a23191cd98047911448d2874871380a31dbb400f0e08cdc2805ce5b5af9ea"
Jul 17 22:34:14.827223 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.827171    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9a23191cd98047911448d2874871380a31dbb400f0e08cdc2805ce5b5af9ea"} err="failed to get container status \"9e9a23191cd98047911448d2874871380a31dbb400f0e08cdc2805ce5b5af9ea\": rpc error: code = NotFound desc = could not find container \"9e9a23191cd98047911448d2874871380a31dbb400f0e08cdc2805ce5b5af9ea\": container with ID starting with 9e9a23191cd98047911448d2874871380a31dbb400f0e08cdc2805ce5b5af9ea not found: ID does not exist"
Jul 17 22:34:14.830322 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.830297    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"]
Jul 17 22:34:14.834445 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:14.834381    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-11aecf-openshift-default-7ddcdf7555-gsnrn"]
Jul 17 22:34:15.183777 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:15.183691    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ba2521a-b4f2-48c9-a99c-ec688f51d605" path="/var/lib/kubelet/pods/2ba2521a-b4f2-48c9-a99c-ec688f51d605/volumes"
Jul 17 22:34:20.199299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:20.199267    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:22.807660 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.807620    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:22.946655 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.946613    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm"]
Jul 17 22:34:22.947015 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.947000    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac788942-8d12-4484-86f6-9861e2b6711f" containerName="revoke-keys"
Jul 17 22:34:22.947096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.947017    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac788942-8d12-4484-86f6-9861e2b6711f" containerName="revoke-keys"
Jul 17 22:34:22.947096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.947029    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2efb5d-9490-43af-ba4d-9e5cde98129e" containerName="revoke-keys"
Jul 17 22:34:22.947096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.947036    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2efb5d-9490-43af-ba4d-9e5cde98129e" containerName="revoke-keys"
Jul 17 22:34:22.947096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.947047    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba2521a-b4f2-48c9-a99c-ec688f51d605" containerName="istio-proxy"
Jul 17 22:34:22.947096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.947053    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba2521a-b4f2-48c9-a99c-ec688f51d605" containerName="istio-proxy"
Jul 17 22:34:22.947096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.947078    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac788942-8d12-4484-86f6-9861e2b6711f" containerName="revoke-keys"
Jul 17 22:34:22.947096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.947084    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac788942-8d12-4484-86f6-9861e2b6711f" containerName="revoke-keys"
Jul 17 22:34:22.947349 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.947139    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba2521a-b4f2-48c9-a99c-ec688f51d605" containerName="istio-proxy"
Jul 17 22:34:22.947349 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.947145    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac788942-8d12-4484-86f6-9861e2b6711f" containerName="revoke-keys"
Jul 17 22:34:22.947349 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.947151    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ba2521a-b4f2-48c9-a99c-ec688f51d605" containerName="istio-proxy"
Jul 17 22:34:22.947349 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.947156    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2efb5d-9490-43af-ba4d-9e5cde98129e" containerName="revoke-keys"
Jul 17 22:34:22.947349 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.947161    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac788942-8d12-4484-86f6-9861e2b6711f" containerName="revoke-keys"
Jul 17 22:34:22.950468 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.950451    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm"
Jul 17 22:34:22.953074 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.953048    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:34:22.953222 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.953057    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:34:22.959845 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:22.959821    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm"]
Jul 17 22:34:23.078399 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:23.078289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4217be1-97fd-4e48-b014-825a7032efa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm\" (UID: \"e4217be1-97fd-4e48-b014-825a7032efa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm"
Jul 17 22:34:23.179039 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:23.179008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4217be1-97fd-4e48-b014-825a7032efa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm\" (UID: \"e4217be1-97fd-4e48-b014-825a7032efa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm"
Jul 17 22:34:23.179632 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:23.179613    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4217be1-97fd-4e48-b014-825a7032efa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm\" (UID: \"e4217be1-97fd-4e48-b014-825a7032efa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm"
Jul 17 22:34:23.262692 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:23.262659    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm"
Jul 17 22:34:23.404980 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:23.404952    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm"]
Jul 17 22:34:23.406845 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:34:23.406819    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4217be1_97fd_4e48_b014_825a7032efa3.slice/crio-24689f6ac957040666cde543ba5f553235ed1df97637010f2a874d2f31b8c7b0 WatchSource:0}: Error finding container 24689f6ac957040666cde543ba5f553235ed1df97637010f2a874d2f31b8c7b0: Status 404 returned error can't find the container with id 24689f6ac957040666cde543ba5f553235ed1df97637010f2a874d2f31b8c7b0
Jul 17 22:34:23.448494 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:23.448460    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:34:23.861688 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:23.861650    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm" event={"ID":"e4217be1-97fd-4e48-b014-825a7032efa3","Type":"ContainerStarted","Data":"411c467a67d1824795f113a148693c1b2cfc730dd7e84425968bc7d74615b57a"}
Jul 17 22:34:23.862118 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:23.861720    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm" event={"ID":"e4217be1-97fd-4e48-b014-825a7032efa3","Type":"ContainerStarted","Data":"24689f6ac957040666cde543ba5f553235ed1df97637010f2a874d2f31b8c7b0"}
Jul 17 22:34:23.878275 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:23.878227    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm" podStartSLOduration=1.8782118410000002 podStartE2EDuration="1.878211841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:34:23.87692419 +0000 UTC m=+2927.265940841" watchObservedRunningTime="2026-07-17 22:34:23.878211841 +0000 UTC m=+2927.267228492"
Jul 17 22:34:24.788339 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:24.788299    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:24.868498 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:24.868458    2565 generic.go:356] "Generic (PLEG): container finished" podID="e4217be1-97fd-4e48-b014-825a7032efa3" containerID="411c467a67d1824795f113a148693c1b2cfc730dd7e84425968bc7d74615b57a" exitCode=6
Jul 17 22:34:24.868901 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:24.868546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm" event={"ID":"e4217be1-97fd-4e48-b014-825a7032efa3","Type":"ContainerDied","Data":"411c467a67d1824795f113a148693c1b2cfc730dd7e84425968bc7d74615b57a"}
Jul 17 22:34:24.868901 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:24.868736    2565 scope.go:117] "RemoveContainer" containerID="411c467a67d1824795f113a148693c1b2cfc730dd7e84425968bc7d74615b57a"
Jul 17 22:34:25.875254 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:25.875217    2565 generic.go:356] "Generic (PLEG): container finished" podID="e4217be1-97fd-4e48-b014-825a7032efa3" containerID="30cbe72a3198a637b46848664d55d35830586deabcd42655ca216ab59ce56b31" exitCode=6
Jul 17 22:34:25.875716 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:25.875302    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm" event={"ID":"e4217be1-97fd-4e48-b014-825a7032efa3","Type":"ContainerDied","Data":"30cbe72a3198a637b46848664d55d35830586deabcd42655ca216ab59ce56b31"}
Jul 17 22:34:25.875716 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:25.875355    2565 scope.go:117] "RemoveContainer" containerID="411c467a67d1824795f113a148693c1b2cfc730dd7e84425968bc7d74615b57a"
Jul 17 22:34:25.875716 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:25.875488    2565 scope.go:117] "RemoveContainer" containerID="30cbe72a3198a637b46848664d55d35830586deabcd42655ca216ab59ce56b31"
Jul 17 22:34:25.875716 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:34:25.875702    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm_odh-ai-gateway-infra(e4217be1-97fd-4e48-b014-825a7032efa3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm" podUID="e4217be1-97fd-4e48-b014-825a7032efa3"
Jul 17 22:34:26.882564 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:26.882534    2565 scope.go:117] "RemoveContainer" containerID="30cbe72a3198a637b46848664d55d35830586deabcd42655ca216ab59ce56b31"
Jul 17 22:34:26.882951 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:34:26.882763    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm_odh-ai-gateway-infra(e4217be1-97fd-4e48-b014-825a7032efa3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm" podUID="e4217be1-97fd-4e48-b014-825a7032efa3"
Jul 17 22:34:30.858644 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:30.858605    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:30.930856 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:30.930825    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:35.546439 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:35.546406    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"]
Jul 17 22:34:35.546905 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:35.546610    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs" podUID="6ebc939d-7c34-4aa1-be7b-90801d6f536c" containerName="istio-proxy" containerID="cri-o://f1d0420b1a45d9a7e09c3442b9fe016aef5432acb09739797ad12ab1a317ec55" gracePeriod=30
Jul 17 22:34:35.578621 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:35.578585    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:35.812366 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:35.812269    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:36.143115 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.143014    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:36.506008 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.505973    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"]
Jul 17 22:34:36.506445 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.506430    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac788942-8d12-4484-86f6-9861e2b6711f" containerName="revoke-keys"
Jul 17 22:34:36.506491 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.506447    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac788942-8d12-4484-86f6-9861e2b6711f" containerName="revoke-keys"
Jul 17 22:34:36.506491 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.506465    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ba2521a-b4f2-48c9-a99c-ec688f51d605" containerName="istio-proxy"
Jul 17 22:34:36.506491 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.506471    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ba2521a-b4f2-48c9-a99c-ec688f51d605" containerName="istio-proxy"
Jul 17 22:34:36.506586 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.506525    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac788942-8d12-4484-86f6-9861e2b6711f" containerName="revoke-keys"
Jul 17 22:34:36.510911 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.510889    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.513784 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.513757    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5abe32-openshift-default-dockercfg-qffzs\""
Jul 17 22:34:36.531976 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.531937    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"]
Jul 17 22:34:36.543543 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.543516    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:36.609815 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.609661    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-data\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.609815 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.609722    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istiod-ca-cert\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.609815 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.609754    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-credential-socket\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.610437 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.609919    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-workload-socket\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.610437 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.610006    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9kd4\" (UniqueName: \"kubernetes.io/projected/0edba988-bc02-4ed9-80ea-982f5fb3e5af-kube-api-access-h9kd4\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.610437 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.610089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-workload-certs\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.610437 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.610122    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-envoy\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.610437 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.610146    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-token\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.610437 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.610179    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-podinfo\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.711040 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.710987    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-workload-socket\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.711277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.711056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9kd4\" (UniqueName: \"kubernetes.io/projected/0edba988-bc02-4ed9-80ea-982f5fb3e5af-kube-api-access-h9kd4\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.711277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.711145    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-workload-certs\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.711277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.711176    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-envoy\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.711277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.711199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-token\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.711277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.711228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-podinfo\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.711277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.711266    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-data\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.711582 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.711319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istiod-ca-cert\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.711582 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.711350    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-credential-socket\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.711685 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.711664    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-workload-certs\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.711734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.711675    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-workload-socket\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.711959 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.711918    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-data\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.712130 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.712034    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-credential-socket\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.712195 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.712158    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istiod-ca-cert\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.713653 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.713631    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-envoy\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.713971 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.713950    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-podinfo\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.721734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.721705    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9kd4\" (UniqueName: \"kubernetes.io/projected/0edba988-bc02-4ed9-80ea-982f5fb3e5af-kube-api-access-h9kd4\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.723095 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.723045    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-token\") pod \"e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") " pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.821537 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.821437    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:36.992319 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.992287    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"]
Jul 17 22:34:36.994209 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:34:36.994185    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0edba988_bc02_4ed9_80ea_982f5fb3e5af.slice/crio-b52cb07c9f74a253b911aff6e333de9d109bd755d92892874f8d6e48e8de441e WatchSource:0}: Error finding container b52cb07c9f74a253b911aff6e333de9d109bd755d92892874f8d6e48e8de441e: Status 404 returned error can't find the container with id b52cb07c9f74a253b911aff6e333de9d109bd755d92892874f8d6e48e8de441e
Jul 17 22:34:36.996482 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.996445    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:34:36.996585 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.996516    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:34:36.996585 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:36.996546    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:34:37.117507 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:37.117408    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:37.940683 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:37.940597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s" event={"ID":"0edba988-bc02-4ed9-80ea-982f5fb3e5af","Type":"ContainerStarted","Data":"948a16e8d73668e89bc9656331dec7cc96e9d3b220bc472c4fd2cd2de4fcf713"}
Jul 17 22:34:37.940683 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:37.940649    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s" event={"ID":"0edba988-bc02-4ed9-80ea-982f5fb3e5af","Type":"ContainerStarted","Data":"b52cb07c9f74a253b911aff6e333de9d109bd755d92892874f8d6e48e8de441e"}
Jul 17 22:34:37.967031 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:37.966975    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s" podStartSLOduration=1.966956926 podStartE2EDuration="1.966956926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:34:37.963565864 +0000 UTC m=+2941.352582514" watchObservedRunningTime="2026-07-17 22:34:37.966956926 +0000 UTC m=+2941.355973576"
Jul 17 22:34:38.176790 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:38.176758    2565 scope.go:117] "RemoveContainer" containerID="30cbe72a3198a637b46848664d55d35830586deabcd42655ca216ab59ce56b31"
Jul 17 22:34:38.822490 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:38.822450    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:38.827583 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:38.827556    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:38.947307 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:38.947271    2565 generic.go:356] "Generic (PLEG): container finished" podID="e4217be1-97fd-4e48-b014-825a7032efa3" containerID="0d86d4b3f51a28aff0295f146b74eadaa502fde6f6561bc9c013aa2ec671bc63" exitCode=6
Jul 17 22:34:38.947795 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:38.947313    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm" event={"ID":"e4217be1-97fd-4e48-b014-825a7032efa3","Type":"ContainerDied","Data":"0d86d4b3f51a28aff0295f146b74eadaa502fde6f6561bc9c013aa2ec671bc63"}
Jul 17 22:34:38.947795 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:38.947380    2565 scope.go:117] "RemoveContainer" containerID="30cbe72a3198a637b46848664d55d35830586deabcd42655ca216ab59ce56b31"
Jul 17 22:34:38.947795 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:38.947673    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:38.947968 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:38.947802    2565 scope.go:117] "RemoveContainer" containerID="0d86d4b3f51a28aff0295f146b74eadaa502fde6f6561bc9c013aa2ec671bc63"
Jul 17 22:34:38.948123 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:34:38.948100    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm_odh-ai-gateway-infra(e4217be1-97fd-4e48-b014-825a7032efa3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm" podUID="e4217be1-97fd-4e48-b014-825a7032efa3"
Jul 17 22:34:38.948958 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:38.948916    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:39.937673 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:39.937631    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:39.980132 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:39.980092    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm"]
Jul 17 22:34:40.040327 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.040300    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:40.119866 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.119842    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm"
Jul 17 22:34:40.142199 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.142167    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4217be1-97fd-4e48-b014-825a7032efa3-maas-api-service-ca\") pod \"e4217be1-97fd-4e48-b014-825a7032efa3\" (UID: \"e4217be1-97fd-4e48-b014-825a7032efa3\") "
Jul 17 22:34:40.142670 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.142626    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4217be1-97fd-4e48-b014-825a7032efa3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e4217be1-97fd-4e48-b014-825a7032efa3" (UID: "e4217be1-97fd-4e48-b014-825a7032efa3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:34:40.215667 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.215570    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:40.244213 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.244171    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4217be1-97fd-4e48-b014-825a7032efa3-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:40.315942 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.315902    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:40.768712 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.768684    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:40.804554 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.804528    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:34:40.838941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.838848    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:40.851304 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.851268    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istiod-ca-cert\") pod \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") "
Jul 17 22:34:40.851448 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.851341    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-workload-certs\") pod \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") "
Jul 17 22:34:40.851448 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.851423    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-envoy\") pod \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") "
Jul 17 22:34:40.851535 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.851462    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-workload-socket\") pod \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") "
Jul 17 22:34:40.851535 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.851504    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxb7s\" (UniqueName: \"kubernetes.io/projected/6ebc939d-7c34-4aa1-be7b-90801d6f536c-kube-api-access-vxb7s\") pod \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") "
Jul 17 22:34:40.851640 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.851544    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-token\") pod \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") "
Jul 17 22:34:40.851640 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.851584    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-podinfo\") pod \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") "
Jul 17 22:34:40.851640 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.851610    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-data\") pod \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") "
Jul 17 22:34:40.851640 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.851626    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ebc939d-7c34-4aa1-be7b-90801d6f536c" (UID: "6ebc939d-7c34-4aa1-be7b-90801d6f536c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:34:40.851844 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.851689    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ebc939d-7c34-4aa1-be7b-90801d6f536c" (UID: "6ebc939d-7c34-4aa1-be7b-90801d6f536c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:34:40.851844 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.851751    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ebc939d-7c34-4aa1-be7b-90801d6f536c" (UID: "6ebc939d-7c34-4aa1-be7b-90801d6f536c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:34:40.851844 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.851803    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-credential-socket\") pod \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\" (UID: \"6ebc939d-7c34-4aa1-be7b-90801d6f536c\") "
Jul 17 22:34:40.852159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.852128    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ebc939d-7c34-4aa1-be7b-90801d6f536c" (UID: "6ebc939d-7c34-4aa1-be7b-90801d6f536c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:34:40.852279 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.852161    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istiod-ca-cert\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:40.852279 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.852182    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-workload-certs\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:40.852279 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.852194    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-workload-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:40.852522 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.852495    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ebc939d-7c34-4aa1-be7b-90801d6f536c" (UID: "6ebc939d-7c34-4aa1-be7b-90801d6f536c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:34:40.854143 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.854118    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebc939d-7c34-4aa1-be7b-90801d6f536c-kube-api-access-vxb7s" (OuterVolumeSpecName: "kube-api-access-vxb7s") pod "6ebc939d-7c34-4aa1-be7b-90801d6f536c" (UID: "6ebc939d-7c34-4aa1-be7b-90801d6f536c"). InnerVolumeSpecName "kube-api-access-vxb7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:34:40.854402 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.854374    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ebc939d-7c34-4aa1-be7b-90801d6f536c" (UID: "6ebc939d-7c34-4aa1-be7b-90801d6f536c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:34:40.854503 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.854375    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ebc939d-7c34-4aa1-be7b-90801d6f536c" (UID: "6ebc939d-7c34-4aa1-be7b-90801d6f536c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:34:40.854503 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.854469    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ebc939d-7c34-4aa1-be7b-90801d6f536c" (UID: "6ebc939d-7c34-4aa1-be7b-90801d6f536c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:34:40.953202 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.953158    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-credential-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:40.953202 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.953196    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-envoy\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:40.953202 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.953209    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxb7s\" (UniqueName: \"kubernetes.io/projected/6ebc939d-7c34-4aa1-be7b-90801d6f536c-kube-api-access-vxb7s\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:40.953458 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.953222    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-token\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:40.953458 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.953234    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-podinfo\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:40.953458 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.953244    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ebc939d-7c34-4aa1-be7b-90801d6f536c-istio-data\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:34:40.960714 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.960684    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm"
Jul 17 22:34:40.960903 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.960713    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm" event={"ID":"e4217be1-97fd-4e48-b014-825a7032efa3","Type":"ContainerDied","Data":"24689f6ac957040666cde543ba5f553235ed1df97637010f2a874d2f31b8c7b0"}
Jul 17 22:34:40.960903 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.960773    2565 scope.go:117] "RemoveContainer" containerID="0d86d4b3f51a28aff0295f146b74eadaa502fde6f6561bc9c013aa2ec671bc63"
Jul 17 22:34:40.962631 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.962602    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ebc939d-7c34-4aa1-be7b-90801d6f536c" containerID="f1d0420b1a45d9a7e09c3442b9fe016aef5432acb09739797ad12ab1a317ec55" exitCode=0
Jul 17 22:34:40.962735 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.962671    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs" event={"ID":"6ebc939d-7c34-4aa1-be7b-90801d6f536c","Type":"ContainerDied","Data":"f1d0420b1a45d9a7e09c3442b9fe016aef5432acb09739797ad12ab1a317ec55"}
Jul 17 22:34:40.962735 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.962709    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"
Jul 17 22:34:40.962735 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.962730    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs" event={"ID":"6ebc939d-7c34-4aa1-be7b-90801d6f536c","Type":"ContainerDied","Data":"f3787647cd8eb266330c57dad36d9f24bf9cc39c3e1bc60181d4b9715656e006"}
Jul 17 22:34:40.970232 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.970154    2565 scope.go:117] "RemoveContainer" containerID="f1d0420b1a45d9a7e09c3442b9fe016aef5432acb09739797ad12ab1a317ec55"
Jul 17 22:34:40.978881 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.978861    2565 scope.go:117] "RemoveContainer" containerID="e0e62752dd23754746afd4c0a2ea6500449cf91b7f108b89b01b25503c193237"
Jul 17 22:34:40.987622 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.987478    2565 scope.go:117] "RemoveContainer" containerID="f1d0420b1a45d9a7e09c3442b9fe016aef5432acb09739797ad12ab1a317ec55"
Jul 17 22:34:40.987856 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:34:40.987751    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d0420b1a45d9a7e09c3442b9fe016aef5432acb09739797ad12ab1a317ec55\": container with ID starting with f1d0420b1a45d9a7e09c3442b9fe016aef5432acb09739797ad12ab1a317ec55 not found: ID does not exist" containerID="f1d0420b1a45d9a7e09c3442b9fe016aef5432acb09739797ad12ab1a317ec55"
Jul 17 22:34:40.987856 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.987779    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d0420b1a45d9a7e09c3442b9fe016aef5432acb09739797ad12ab1a317ec55"} err="failed to get container status \"f1d0420b1a45d9a7e09c3442b9fe016aef5432acb09739797ad12ab1a317ec55\": rpc error: code = NotFound desc = could not find container \"f1d0420b1a45d9a7e09c3442b9fe016aef5432acb09739797ad12ab1a317ec55\": container with ID starting with f1d0420b1a45d9a7e09c3442b9fe016aef5432acb09739797ad12ab1a317ec55 not found: ID does not exist"
Jul 17 22:34:40.987856 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.987796    2565 scope.go:117] "RemoveContainer" containerID="e0e62752dd23754746afd4c0a2ea6500449cf91b7f108b89b01b25503c193237"
Jul 17 22:34:40.988053 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:34:40.988031    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e62752dd23754746afd4c0a2ea6500449cf91b7f108b89b01b25503c193237\": container with ID starting with e0e62752dd23754746afd4c0a2ea6500449cf91b7f108b89b01b25503c193237 not found: ID does not exist" containerID="e0e62752dd23754746afd4c0a2ea6500449cf91b7f108b89b01b25503c193237"
Jul 17 22:34:40.988112 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.988060    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e62752dd23754746afd4c0a2ea6500449cf91b7f108b89b01b25503c193237"} err="failed to get container status \"e0e62752dd23754746afd4c0a2ea6500449cf91b7f108b89b01b25503c193237\": rpc error: code = NotFound desc = could not find container \"e0e62752dd23754746afd4c0a2ea6500449cf91b7f108b89b01b25503c193237\": container with ID starting with e0e62752dd23754746afd4c0a2ea6500449cf91b7f108b89b01b25503c193237 not found: ID does not exist"
Jul 17 22:34:40.988686 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:40.988669    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"]
Jul 17 22:34:41.001584 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:41.001559    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4ccee5-openshift-default-7dd4d55dc7-tzdvs"]
Jul 17 22:34:41.017636 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:41.017603    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm"]
Jul 17 22:34:41.023504 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:41.023474    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-hr2fm"]
Jul 17 22:34:41.182835 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:41.182797    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ebc939d-7c34-4aa1-be7b-90801d6f536c" path="/var/lib/kubelet/pods/6ebc939d-7c34-4aa1-be7b-90801d6f536c/volumes"
Jul 17 22:34:41.183185 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:41.183171    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4217be1-97fd-4e48-b014-825a7032efa3" path="/var/lib/kubelet/pods/e4217be1-97fd-4e48-b014-825a7032efa3/volumes"
Jul 17 22:34:44.242482 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:44.240894    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:34:44.407814 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:44.407780    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:44.874519 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:44.874473    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:45.176153 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.175857    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"]
Jul 17 22:34:45.176421 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.176398    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ebc939d-7c34-4aa1-be7b-90801d6f536c" containerName="istio-proxy"
Jul 17 22:34:45.176577 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.176564    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ebc939d-7c34-4aa1-be7b-90801d6f536c" containerName="istio-proxy"
Jul 17 22:34:45.176678 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.176668    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4217be1-97fd-4e48-b014-825a7032efa3" containerName="revoke-keys"
Jul 17 22:34:45.176750 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.176741    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4217be1-97fd-4e48-b014-825a7032efa3" containerName="revoke-keys"
Jul 17 22:34:45.176818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.176809    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ebc939d-7c34-4aa1-be7b-90801d6f536c" containerName="istio-proxy"
Jul 17 22:34:45.176877 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.176870    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ebc939d-7c34-4aa1-be7b-90801d6f536c" containerName="istio-proxy"
Jul 17 22:34:45.176960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.176951    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4217be1-97fd-4e48-b014-825a7032efa3" containerName="revoke-keys"
Jul 17 22:34:45.177026 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.177018    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4217be1-97fd-4e48-b014-825a7032efa3" containerName="revoke-keys"
Jul 17 22:34:45.177131 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.177119    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4217be1-97fd-4e48-b014-825a7032efa3" containerName="revoke-keys"
Jul 17 22:34:45.177207 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.177199    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4217be1-97fd-4e48-b014-825a7032efa3" containerName="revoke-keys"
Jul 17 22:34:45.177399 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.177389    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4217be1-97fd-4e48-b014-825a7032efa3" containerName="revoke-keys"
Jul 17 22:34:45.177485 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.177475    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ebc939d-7c34-4aa1-be7b-90801d6f536c" containerName="istio-proxy"
Jul 17 22:34:45.177567 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.177559    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ebc939d-7c34-4aa1-be7b-90801d6f536c" containerName="istio-proxy"
Jul 17 22:34:45.177639 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.177631    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4217be1-97fd-4e48-b014-825a7032efa3" containerName="revoke-keys"
Jul 17 22:34:45.177711 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.177701    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4217be1-97fd-4e48-b014-825a7032efa3" containerName="revoke-keys"
Jul 17 22:34:45.192458 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.192422    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.196530 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.196224    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bf8fd3-openshift-default-dockercfg-cktrj\""
Jul 17 22:34:45.199181 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.199151    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"]
Jul 17 22:34:45.206004 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.205952    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:45.299112 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.298572    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-workload-socket\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.299112 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.298639    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-podinfo\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.299112 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.298732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9s56\" (UniqueName: \"kubernetes.io/projected/d81b64d0-ded6-4a8e-946f-7f9b41630a98-kube-api-access-g9s56\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.299112 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.298767    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-workload-certs\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.299112 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.298830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-credential-socket\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.299112 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.298864    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-envoy\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.299112 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.298888    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-token\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.299112 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.298926    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istiod-ca-cert\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.299112 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.298974    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-data\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.400307 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.400269    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-credential-socket\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.400307 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.400311    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-envoy\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.400556 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.400330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-token\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.400556 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.400355    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istiod-ca-cert\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.400556 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.400399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-data\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.400556 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.400430    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-workload-socket\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.400556 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.400451    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-podinfo\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.400556 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.400496    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9s56\" (UniqueName: \"kubernetes.io/projected/d81b64d0-ded6-4a8e-946f-7f9b41630a98-kube-api-access-g9s56\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.400556 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.400517    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-workload-certs\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.400923 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.400787    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-credential-socket\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.400923 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.400865    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-workload-certs\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.400923 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.400900    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-workload-socket\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.401266 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.401143    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-data\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.401387 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.401366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istiod-ca-cert\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.402857 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.402831    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-envoy\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.403210 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.403185    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-podinfo\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.413592 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.413566    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9s56\" (UniqueName: \"kubernetes.io/projected/d81b64d0-ded6-4a8e-946f-7f9b41630a98-kube-api-access-g9s56\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.413946 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.413927    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-token\") pod \"e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") " pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.478716 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.478622    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:45.508347 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.508297    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:45.706388 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.706359    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"]
Jul 17 22:34:45.708943 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:34:45.708915    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81b64d0_ded6_4a8e_946f_7f9b41630a98.slice/crio-67affc56ac632477bfbfe886dce573ea3446f1fd54b0996d8202359619cd37c9 WatchSource:0}: Error finding container 67affc56ac632477bfbfe886dce573ea3446f1fd54b0996d8202359619cd37c9: Status 404 returned error can't find the container with id 67affc56ac632477bfbfe886dce573ea3446f1fd54b0996d8202359619cd37c9
Jul 17 22:34:45.710951 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.710913    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:34:45.711045 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.710989    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:34:45.711045 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.711021    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:34:45.995833 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.995801    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd" event={"ID":"d81b64d0-ded6-4a8e-946f-7f9b41630a98","Type":"ContainerStarted","Data":"d44b3bac6e18412cb9bc748ede7daee7e4ee66ef7708c7fcfdf35e5e3a76751d"}
Jul 17 22:34:45.996034 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:45.995861    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd" event={"ID":"d81b64d0-ded6-4a8e-946f-7f9b41630a98","Type":"ContainerStarted","Data":"67affc56ac632477bfbfe886dce573ea3446f1fd54b0996d8202359619cd37c9"}
Jul 17 22:34:46.023030 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:46.022963    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd" podStartSLOduration=1.022946428 podStartE2EDuration="1.022946428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:34:46.020593464 +0000 UTC m=+2949.409610129" watchObservedRunningTime="2026-07-17 22:34:46.022946428 +0000 UTC m=+2949.411963078"
Jul 17 22:34:46.508822 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:46.508780    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:46.514735 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:46.514706    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:47.000707 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:47.000678    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:47.001683 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:47.001660    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:34:47.980248 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:47.980197    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:48.079138 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:48.079103    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:48.164190 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:48.164146    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:48.479513 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:48.479469    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:48.710255 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:48.710205    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:48.810842 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:48.810798    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:52.028161 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:52.028126    2565 generic.go:356] "Generic (PLEG): container finished" podID="0edba988-bc02-4ed9-80ea-982f5fb3e5af" containerID="948a16e8d73668e89bc9656331dec7cc96e9d3b220bc472c4fd2cd2de4fcf713" exitCode=0
Jul 17 22:34:52.028161 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:52.028165    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s" event={"ID":"0edba988-bc02-4ed9-80ea-982f5fb3e5af","Type":"ContainerDied","Data":"948a16e8d73668e89bc9656331dec7cc96e9d3b220bc472c4fd2cd2de4fcf713"}
Jul 17 22:34:52.028650 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:52.028574    2565 scope.go:117] "RemoveContainer" containerID="948a16e8d73668e89bc9656331dec7cc96e9d3b220bc472c4fd2cd2de4fcf713"
Jul 17 22:34:52.029616 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:52.029577    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:34:52.029737 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:52.029660    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:34:52.029737 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:52.029691    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:34:52.419462 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:52.419366    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:34:53.035321 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:53.035286    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s" event={"ID":"0edba988-bc02-4ed9-80ea-982f5fb3e5af","Type":"ContainerStarted","Data":"c885af161cea94e6fb7b9197b1e9cd5ff388252a81509e636cf224f80076a274"}
Jul 17 22:34:53.821805 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:53.821763    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:53.823227 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:53.823198    2565 patch_prober.go:28] interesting pod/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Jul 17 22:34:53.823352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:53.823250    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s" podUID="0edba988-bc02-4ed9-80ea-982f5fb3e5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Jul 17 22:34:53.948708 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:53.948668    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:54.822553 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:54.822510    2565 patch_prober.go:28] interesting pod/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Jul 17 22:34:54.822983 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:54.822582    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s" podUID="0edba988-bc02-4ed9-80ea-982f5fb3e5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Jul 17 22:34:55.822952 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:55.822907    2565 patch_prober.go:28] interesting pod/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Jul 17 22:34:55.823469 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:55.822977    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s" podUID="0edba988-bc02-4ed9-80ea-982f5fb3e5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Jul 17 22:34:56.644591 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:56.644551    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:34:56.822897 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:56.822855    2565 patch_prober.go:28] interesting pod/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Jul 17 22:34:56.823098 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:56.822939    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s" podUID="0edba988-bc02-4ed9-80ea-982f5fb3e5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Jul 17 22:34:57.826289 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:57.826260    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:34:57.827232 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:34:57.827214    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:35:01.039436 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:01.039404    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:01.140183 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:01.140149    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:03.043317 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:03.043272    2565 patch_prober.go:28] interesting pod/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 17 22:35:03.043797 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:03.043359    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd" podUID="d81b64d0-ded6-4a8e-946f-7f9b41630a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 17 22:35:05.111800 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:05.111711    2565 generic.go:356] "Generic (PLEG): container finished" podID="d81b64d0-ded6-4a8e-946f-7f9b41630a98" containerID="d44b3bac6e18412cb9bc748ede7daee7e4ee66ef7708c7fcfdf35e5e3a76751d" exitCode=0
Jul 17 22:35:05.111800 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:05.111779    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd" event={"ID":"d81b64d0-ded6-4a8e-946f-7f9b41630a98","Type":"ContainerDied","Data":"d44b3bac6e18412cb9bc748ede7daee7e4ee66ef7708c7fcfdf35e5e3a76751d"}
Jul 17 22:35:05.112269 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:05.112223    2565 scope.go:117] "RemoveContainer" containerID="d44b3bac6e18412cb9bc748ede7daee7e4ee66ef7708c7fcfdf35e5e3a76751d"
Jul 17 22:35:05.113235 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:05.113201    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:35:05.113310 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:05.113293    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:35:05.113354 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:05.113333    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:35:06.118476 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:06.118439    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd" event={"ID":"d81b64d0-ded6-4a8e-946f-7f9b41630a98","Type":"ContainerStarted","Data":"ab2e0e7bbfd393502338ef6a7277ad354c964158fefc534a720719332182f502"}
Jul 17 22:35:06.509153 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:06.509118    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:35:06.509442 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:06.509421    2565 patch_prober.go:28] interesting pod/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Jul 17 22:35:06.509489 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:06.509468    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd" podUID="d81b64d0-ded6-4a8e-946f-7f9b41630a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Jul 17 22:35:07.509013 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:07.508978    2565 patch_prober.go:28] interesting pod/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Jul 17 22:35:07.509432 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:07.509034    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd" podUID="d81b64d0-ded6-4a8e-946f-7f9b41630a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Jul 17 22:35:08.508841 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:08.508801    2565 patch_prober.go:28] interesting pod/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Jul 17 22:35:08.509030 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:08.508858    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd" podUID="d81b64d0-ded6-4a8e-946f-7f9b41630a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Jul 17 22:35:09.509278 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:09.509233    2565 patch_prober.go:28] interesting pod/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Jul 17 22:35:09.509690 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:09.509292    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd" podUID="d81b64d0-ded6-4a8e-946f-7f9b41630a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Jul 17 22:35:09.616428 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:09.616397    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:09.718059 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:09.718027    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:10.513095 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:10.513051    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:35:10.513548 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:10.513536    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:35:10.514436 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:10.514420    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:35:11.106546 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:11.106509    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:11.207171 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:11.207138    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:11.392537 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:11.392454    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:15.249076 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:15.249030    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:35:15.500571 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:15.500458    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:21.670119 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:21.670057    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:21.758837 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:21.758805    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:26.253713 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:26.253681    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"]
Jul 17 22:35:26.254245 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:26.253896    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s" podUID="0edba988-bc02-4ed9-80ea-982f5fb3e5af" containerName="istio-proxy" containerID="cri-o://c885af161cea94e6fb7b9197b1e9cd5ff388252a81509e636cf224f80076a274" gracePeriod=30
Jul 17 22:35:26.279061 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:26.279030    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:26.491689 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:26.491656    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:27.231811 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:27.231775    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:27.335653 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:27.335621    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:27.829168 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:27.829127    2565 patch_prober.go:28] interesting pod/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 22:35:27.829341 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:27.829205    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s" podUID="0edba988-bc02-4ed9-80ea-982f5fb3e5af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 22:35:31.508200 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.508165    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:35:31.570595 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.570492    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-workload-certs\") pod \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") "
Jul 17 22:35:31.570595 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.570545    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-data\") pod \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") "
Jul 17 22:35:31.570595 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.570576    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9kd4\" (UniqueName: \"kubernetes.io/projected/0edba988-bc02-4ed9-80ea-982f5fb3e5af-kube-api-access-h9kd4\") pod \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") "
Jul 17 22:35:31.570595 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.570596    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-workload-socket\") pod \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") "
Jul 17 22:35:31.570944 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.570634    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istiod-ca-cert\") pod \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") "
Jul 17 22:35:31.570944 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.570750    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-envoy\") pod \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") "
Jul 17 22:35:31.570944 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.570832    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-podinfo\") pod \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") "
Jul 17 22:35:31.570944 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.570850    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0edba988-bc02-4ed9-80ea-982f5fb3e5af" (UID: "0edba988-bc02-4ed9-80ea-982f5fb3e5af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:35:31.570944 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.570864    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-token\") pod \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") "
Jul 17 22:35:31.570944 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.570903    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-credential-socket\") pod \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\" (UID: \"0edba988-bc02-4ed9-80ea-982f5fb3e5af\") "
Jul 17 22:35:31.571291 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.570989    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0edba988-bc02-4ed9-80ea-982f5fb3e5af" (UID: "0edba988-bc02-4ed9-80ea-982f5fb3e5af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:35:31.571291 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.571119    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0edba988-bc02-4ed9-80ea-982f5fb3e5af" (UID: "0edba988-bc02-4ed9-80ea-982f5fb3e5af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:35:31.571569 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.571323    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0edba988-bc02-4ed9-80ea-982f5fb3e5af" (UID: "0edba988-bc02-4ed9-80ea-982f5fb3e5af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:35:31.571569 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.571471    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-credential-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:31.571569 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.571493    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-workload-certs\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:31.571569 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.571507    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-workload-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:31.571569 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.571519    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istiod-ca-cert\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:31.571835 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.571569    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-data" (OuterVolumeSpecName: "istio-data") pod "0edba988-bc02-4ed9-80ea-982f5fb3e5af" (UID: "0edba988-bc02-4ed9-80ea-982f5fb3e5af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:35:31.573130 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.573106    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0edba988-bc02-4ed9-80ea-982f5fb3e5af-kube-api-access-h9kd4" (OuterVolumeSpecName: "kube-api-access-h9kd4") pod "0edba988-bc02-4ed9-80ea-982f5fb3e5af" (UID: "0edba988-bc02-4ed9-80ea-982f5fb3e5af"). InnerVolumeSpecName "kube-api-access-h9kd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:35:31.573313 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.573295    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0edba988-bc02-4ed9-80ea-982f5fb3e5af" (UID: "0edba988-bc02-4ed9-80ea-982f5fb3e5af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:35:31.573364 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.573316    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0edba988-bc02-4ed9-80ea-982f5fb3e5af" (UID: "0edba988-bc02-4ed9-80ea-982f5fb3e5af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:35:31.573364 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.573316    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-token" (OuterVolumeSpecName: "istio-token") pod "0edba988-bc02-4ed9-80ea-982f5fb3e5af" (UID: "0edba988-bc02-4ed9-80ea-982f5fb3e5af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:35:31.672869 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.672829    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-data\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:31.672869 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.672861    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9kd4\" (UniqueName: \"kubernetes.io/projected/0edba988-bc02-4ed9-80ea-982f5fb3e5af-kube-api-access-h9kd4\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:31.672869 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.672875    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-envoy\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:31.673197 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.672887    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-podinfo\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:31.673197 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:31.672897    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0edba988-bc02-4ed9-80ea-982f5fb3e5af-istio-token\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:32.251579 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:32.251545    2565 generic.go:356] "Generic (PLEG): container finished" podID="0edba988-bc02-4ed9-80ea-982f5fb3e5af" containerID="c885af161cea94e6fb7b9197b1e9cd5ff388252a81509e636cf224f80076a274" exitCode=0
Jul 17 22:35:32.251768 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:32.251597    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"
Jul 17 22:35:32.251768 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:32.251633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s" event={"ID":"0edba988-bc02-4ed9-80ea-982f5fb3e5af","Type":"ContainerDied","Data":"c885af161cea94e6fb7b9197b1e9cd5ff388252a81509e636cf224f80076a274"}
Jul 17 22:35:32.251768 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:32.251692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s" event={"ID":"0edba988-bc02-4ed9-80ea-982f5fb3e5af","Type":"ContainerDied","Data":"b52cb07c9f74a253b911aff6e333de9d109bd755d92892874f8d6e48e8de441e"}
Jul 17 22:35:32.251768 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:32.251724    2565 scope.go:117] "RemoveContainer" containerID="c885af161cea94e6fb7b9197b1e9cd5ff388252a81509e636cf224f80076a274"
Jul 17 22:35:32.261416 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:32.261396    2565 scope.go:117] "RemoveContainer" containerID="948a16e8d73668e89bc9656331dec7cc96e9d3b220bc472c4fd2cd2de4fcf713"
Jul 17 22:35:32.270201 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:32.270173    2565 scope.go:117] "RemoveContainer" containerID="c885af161cea94e6fb7b9197b1e9cd5ff388252a81509e636cf224f80076a274"
Jul 17 22:35:32.270538 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:35:32.270516    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c885af161cea94e6fb7b9197b1e9cd5ff388252a81509e636cf224f80076a274\": container with ID starting with c885af161cea94e6fb7b9197b1e9cd5ff388252a81509e636cf224f80076a274 not found: ID does not exist" containerID="c885af161cea94e6fb7b9197b1e9cd5ff388252a81509e636cf224f80076a274"
Jul 17 22:35:32.270594 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:32.270549    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c885af161cea94e6fb7b9197b1e9cd5ff388252a81509e636cf224f80076a274"} err="failed to get container status \"c885af161cea94e6fb7b9197b1e9cd5ff388252a81509e636cf224f80076a274\": rpc error: code = NotFound desc = could not find container \"c885af161cea94e6fb7b9197b1e9cd5ff388252a81509e636cf224f80076a274\": container with ID starting with c885af161cea94e6fb7b9197b1e9cd5ff388252a81509e636cf224f80076a274 not found: ID does not exist"
Jul 17 22:35:32.270594 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:32.270571    2565 scope.go:117] "RemoveContainer" containerID="948a16e8d73668e89bc9656331dec7cc96e9d3b220bc472c4fd2cd2de4fcf713"
Jul 17 22:35:32.270841 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:35:32.270824    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"948a16e8d73668e89bc9656331dec7cc96e9d3b220bc472c4fd2cd2de4fcf713\": container with ID starting with 948a16e8d73668e89bc9656331dec7cc96e9d3b220bc472c4fd2cd2de4fcf713 not found: ID does not exist" containerID="948a16e8d73668e89bc9656331dec7cc96e9d3b220bc472c4fd2cd2de4fcf713"
Jul 17 22:35:32.270892 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:32.270846    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"948a16e8d73668e89bc9656331dec7cc96e9d3b220bc472c4fd2cd2de4fcf713"} err="failed to get container status \"948a16e8d73668e89bc9656331dec7cc96e9d3b220bc472c4fd2cd2de4fcf713\": rpc error: code = NotFound desc = could not find container \"948a16e8d73668e89bc9656331dec7cc96e9d3b220bc472c4fd2cd2de4fcf713\": container with ID starting with 948a16e8d73668e89bc9656331dec7cc96e9d3b220bc472c4fd2cd2de4fcf713 not found: ID does not exist"
Jul 17 22:35:32.280377 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:32.280344    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"]
Jul 17 22:35:32.285238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:32.285211    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5abe32-openshift-default-5ffc4f85dc-xwx5s"]
Jul 17 22:35:32.384783 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:32.384733    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:32.482603 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:32.482563    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:33.182508 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:33.182476    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0edba988-bc02-4ed9-80ea-982f5fb3e5af" path="/var/lib/kubelet/pods/0edba988-bc02-4ed9-80ea-982f5fb3e5af/volumes"
Jul 17 22:35:37.205281 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:37.205239    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:37.307638 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:37.307606    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:37.325481 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:37.325458    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:35:37.340372 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:37.340344    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:35:37.497094 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:37.497034    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:40.652737 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:40.652698    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:35:41.696291 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:41.696256    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:47.781584 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:47.781552    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:47.876373 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:47.876342    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:48.453626 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:48.453589    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456"]
Jul 17 22:35:48.453971 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:48.453958    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0edba988-bc02-4ed9-80ea-982f5fb3e5af" containerName="istio-proxy"
Jul 17 22:35:48.454016 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:48.453972    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0edba988-bc02-4ed9-80ea-982f5fb3e5af" containerName="istio-proxy"
Jul 17 22:35:48.454016 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:48.453983    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0edba988-bc02-4ed9-80ea-982f5fb3e5af" containerName="istio-proxy"
Jul 17 22:35:48.454016 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:48.453989    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0edba988-bc02-4ed9-80ea-982f5fb3e5af" containerName="istio-proxy"
Jul 17 22:35:48.454170 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:48.454043    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0edba988-bc02-4ed9-80ea-982f5fb3e5af" containerName="istio-proxy"
Jul 17 22:35:48.454170 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:48.454050    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0edba988-bc02-4ed9-80ea-982f5fb3e5af" containerName="istio-proxy"
Jul 17 22:35:48.457177 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:48.457156    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456"
Jul 17 22:35:48.463740 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:48.463719    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:35:48.464043 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:48.464020    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:35:48.467444 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:48.467420    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456"]
Jul 17 22:35:48.532688 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:48.532651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423dca54-e885-4c31-ac52-805543e0ee5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-cv456\" (UID: \"423dca54-e885-4c31-ac52-805543e0ee5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456"
Jul 17 22:35:48.633983 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:48.633939    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423dca54-e885-4c31-ac52-805543e0ee5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-cv456\" (UID: \"423dca54-e885-4c31-ac52-805543e0ee5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456"
Jul 17 22:35:48.634677 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:48.634638    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423dca54-e885-4c31-ac52-805543e0ee5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-cv456\" (UID: \"423dca54-e885-4c31-ac52-805543e0ee5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456"
Jul 17 22:35:48.768256 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:48.768218    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456"
Jul 17 22:35:48.909249 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:48.909220    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456"]
Jul 17 22:35:48.911342 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:35:48.911308    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423dca54_e885_4c31_ac52_805543e0ee5e.slice/crio-8f3ea29c8a217c0263367a27d2402e0b6b0bdc61c68ed08b303918db526f6d62 WatchSource:0}: Error finding container 8f3ea29c8a217c0263367a27d2402e0b6b0bdc61c68ed08b303918db526f6d62: Status 404 returned error can't find the container with id 8f3ea29c8a217c0263367a27d2402e0b6b0bdc61c68ed08b303918db526f6d62
Jul 17 22:35:49.336098 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:49.335981    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456" event={"ID":"423dca54-e885-4c31-ac52-805543e0ee5e","Type":"ContainerStarted","Data":"0144fe26df3ee232799521e6ade8c5e071a63ae078f55dcf13918d069251dd01"}
Jul 17 22:35:49.336098 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:49.336038    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456" event={"ID":"423dca54-e885-4c31-ac52-805543e0ee5e","Type":"ContainerStarted","Data":"8f3ea29c8a217c0263367a27d2402e0b6b0bdc61c68ed08b303918db526f6d62"}
Jul 17 22:35:49.355013 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:49.354956    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456" podStartSLOduration=1.354936667 podStartE2EDuration="1.354936667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:35:49.353458504 +0000 UTC m=+3012.742475175" watchObservedRunningTime="2026-07-17 22:35:49.354936667 +0000 UTC m=+3012.743953318"
Jul 17 22:35:50.342203 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:50.342166    2565 generic.go:356] "Generic (PLEG): container finished" podID="423dca54-e885-4c31-ac52-805543e0ee5e" containerID="0144fe26df3ee232799521e6ade8c5e071a63ae078f55dcf13918d069251dd01" exitCode=6
Jul 17 22:35:50.342698 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:50.342256    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456" event={"ID":"423dca54-e885-4c31-ac52-805543e0ee5e","Type":"ContainerDied","Data":"0144fe26df3ee232799521e6ade8c5e071a63ae078f55dcf13918d069251dd01"}
Jul 17 22:35:50.342698 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:50.342494    2565 scope.go:117] "RemoveContainer" containerID="0144fe26df3ee232799521e6ade8c5e071a63ae078f55dcf13918d069251dd01"
Jul 17 22:35:51.349318 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:51.349284    2565 generic.go:356] "Generic (PLEG): container finished" podID="423dca54-e885-4c31-ac52-805543e0ee5e" containerID="e219ea66f759ca47ccdd92f67cc7fbc5767efe792079085c10bbf0d576d909b9" exitCode=6
Jul 17 22:35:51.349793 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:51.349336    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456" event={"ID":"423dca54-e885-4c31-ac52-805543e0ee5e","Type":"ContainerDied","Data":"e219ea66f759ca47ccdd92f67cc7fbc5767efe792079085c10bbf0d576d909b9"}
Jul 17 22:35:51.349793 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:51.349397    2565 scope.go:117] "RemoveContainer" containerID="0144fe26df3ee232799521e6ade8c5e071a63ae078f55dcf13918d069251dd01"
Jul 17 22:35:51.349793 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:51.349556    2565 scope.go:117] "RemoveContainer" containerID="e219ea66f759ca47ccdd92f67cc7fbc5767efe792079085c10bbf0d576d909b9"
Jul 17 22:35:51.349920 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:35:51.349822    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-cv456_odh-ai-gateway-infra(423dca54-e885-4c31-ac52-805543e0ee5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456" podUID="423dca54-e885-4c31-ac52-805543e0ee5e"
Jul 17 22:35:52.357784 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:52.357759    2565 scope.go:117] "RemoveContainer" containerID="e219ea66f759ca47ccdd92f67cc7fbc5767efe792079085c10bbf0d576d909b9"
Jul 17 22:35:52.358311 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:35:52.358014    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-cv456_odh-ai-gateway-infra(423dca54-e885-4c31-ac52-805543e0ee5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456" podUID="423dca54-e885-4c31-ac52-805543e0ee5e"
Jul 17 22:35:52.365653 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:52.365618    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"]
Jul 17 22:35:52.365873 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:52.365850    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd" podUID="d81b64d0-ded6-4a8e-946f-7f9b41630a98" containerName="istio-proxy" containerID="cri-o://ab2e0e7bbfd393502338ef6a7277ad354c964158fefc534a720719332182f502" gracePeriod=30
Jul 17 22:35:52.385350 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:52.385318    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:52.621748 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:52.621659    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:53.000669 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.000639    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:53.370583 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.370494    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"]
Jul 17 22:35:53.375200 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.375180    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.378959 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.378939    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-a56653-openshift-default-dockercfg-bfkh8\""
Jul 17 22:35:53.402354 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.402327    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"]
Jul 17 22:35:53.438895 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.438853    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:53.480906 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.480870    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-podinfo\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.481121 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.480911    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-data\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.481121 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.480935    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-workload-socket\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.481121 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.480959    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722b2832-2199-480b-ac12-f84cc1beb2dc-istiod-ca-cert\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.481121 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.481046    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-envoy\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.481121 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.481107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsq52\" (UniqueName: \"kubernetes.io/projected/722b2832-2199-480b-ac12-f84cc1beb2dc-kube-api-access-wsq52\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.481420 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.481139    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-workload-certs\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.481420 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.481162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-credential-socket\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.481420 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.481283    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-token\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.581954 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.581911    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-envoy\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.581954 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.581954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsq52\" (UniqueName: \"kubernetes.io/projected/722b2832-2199-480b-ac12-f84cc1beb2dc-kube-api-access-wsq52\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.582247 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.581974    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-workload-certs\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.582247 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.582002    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-credential-socket\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.582247 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.582100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-token\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.582247 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.582198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-podinfo\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.582247 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.582228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-data\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.582515 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.582255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-workload-socket\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.582515 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.582282    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722b2832-2199-480b-ac12-f84cc1beb2dc-istiod-ca-cert\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.582515 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.582418    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-workload-certs\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.582644 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.582520    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-credential-socket\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.582719 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.582700    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-workload-socket\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.582768 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.582723    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-data\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.582959 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.582939    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722b2832-2199-480b-ac12-f84cc1beb2dc-istiod-ca-cert\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.584840 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.584812    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-envoy\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.585192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.585168    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-podinfo\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.591853 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.591824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsq52\" (UniqueName: \"kubernetes.io/projected/722b2832-2199-480b-ac12-f84cc1beb2dc-kube-api-access-wsq52\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.591981 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.591814    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-token\") pod \"e2e-shared-a-a56653-openshift-default-6786985bff-njl8x\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") " pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.686661 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.686575    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:53.837584 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.837556    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"]
Jul 17 22:35:53.839784 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:35:53.839753    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722b2832_2199_480b_ac12_f84cc1beb2dc.slice/crio-bed0c56aba1aaa2bfac0693309201e7da58c12803fd62c75a110c1b7c01f2976 WatchSource:0}: Error finding container bed0c56aba1aaa2bfac0693309201e7da58c12803fd62c75a110c1b7c01f2976: Status 404 returned error can't find the container with id bed0c56aba1aaa2bfac0693309201e7da58c12803fd62c75a110c1b7c01f2976
Jul 17 22:35:53.842228 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.842186    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:35:53.842316 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.842302    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:35:53.842361 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:53.842346    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:35:54.255722 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:54.255690    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:54.370018 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:54.369977    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x" event={"ID":"722b2832-2199-480b-ac12-f84cc1beb2dc","Type":"ContainerStarted","Data":"3fd9fe91b8f0ec045d01a0712bba2f2293edc91e278710af560e1dfb72befdc0"}
Jul 17 22:35:54.370260 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:54.370058    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x" event={"ID":"722b2832-2199-480b-ac12-f84cc1beb2dc","Type":"ContainerStarted","Data":"bed0c56aba1aaa2bfac0693309201e7da58c12803fd62c75a110c1b7c01f2976"}
Jul 17 22:35:54.395281 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:54.395225    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x" podStartSLOduration=1.395207931 podStartE2EDuration="1.395207931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:35:54.392356943 +0000 UTC m=+3017.781373593" watchObservedRunningTime="2026-07-17 22:35:54.395207931 +0000 UTC m=+3017.784224582"
Jul 17 22:35:54.687128 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:54.687019    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:54.692336 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:54.692309    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:55.374642 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:55.374605    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:55.375723 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:55.375687    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:35:55.516457 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:55.516414    2565 patch_prober.go:28] interesting pod/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 22:35:55.516864 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:55.516484    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd" podUID="d81b64d0-ded6-4a8e-946f-7f9b41630a98" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 22:35:56.351107 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:56.351054    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:56.453351 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:56.453308    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:56.597479 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:56.597449    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:56.856603 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:56.856567    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:57.092572 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.092545    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:57.193336 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.193245    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:57.633187 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.633152    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:35:57.723199 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.723168    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-envoy\") pod \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") "
Jul 17 22:35:57.723199 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.723205    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-workload-socket\") pod \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") "
Jul 17 22:35:57.723449 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.723253    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-token\") pod \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") "
Jul 17 22:35:57.723449 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.723285    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-data\") pod \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") "
Jul 17 22:35:57.723449 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.723318    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-podinfo\") pod \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") "
Jul 17 22:35:57.723449 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.723358    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istiod-ca-cert\") pod \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") "
Jul 17 22:35:57.723449 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.723399    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-credential-socket\") pod \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") "
Jul 17 22:35:57.723706 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.723459    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9s56\" (UniqueName: \"kubernetes.io/projected/d81b64d0-ded6-4a8e-946f-7f9b41630a98-kube-api-access-g9s56\") pod \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") "
Jul 17 22:35:57.723706 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.723484    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-workload-certs\") pod \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\" (UID: \"d81b64d0-ded6-4a8e-946f-7f9b41630a98\") "
Jul 17 22:35:57.724137 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.723537    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d81b64d0-ded6-4a8e-946f-7f9b41630a98" (UID: "d81b64d0-ded6-4a8e-946f-7f9b41630a98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:35:57.724137 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.723735    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d81b64d0-ded6-4a8e-946f-7f9b41630a98" (UID: "d81b64d0-ded6-4a8e-946f-7f9b41630a98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:35:57.724137 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.723817    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d81b64d0-ded6-4a8e-946f-7f9b41630a98" (UID: "d81b64d0-ded6-4a8e-946f-7f9b41630a98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:35:57.724137 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.724010    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d81b64d0-ded6-4a8e-946f-7f9b41630a98" (UID: "d81b64d0-ded6-4a8e-946f-7f9b41630a98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:35:57.724397 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.724293    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-workload-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:57.724397 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.724313    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istiod-ca-cert\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:57.724397 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.724327    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-credential-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:57.724397 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.724341    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-workload-certs\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:57.724555 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.724529    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-data" (OuterVolumeSpecName: "istio-data") pod "d81b64d0-ded6-4a8e-946f-7f9b41630a98" (UID: "d81b64d0-ded6-4a8e-946f-7f9b41630a98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:35:57.725686 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.725658    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d81b64d0-ded6-4a8e-946f-7f9b41630a98" (UID: "d81b64d0-ded6-4a8e-946f-7f9b41630a98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:35:57.725819 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.725706    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-token" (OuterVolumeSpecName: "istio-token") pod "d81b64d0-ded6-4a8e-946f-7f9b41630a98" (UID: "d81b64d0-ded6-4a8e-946f-7f9b41630a98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:35:57.726102 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.726056    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d81b64d0-ded6-4a8e-946f-7f9b41630a98" (UID: "d81b64d0-ded6-4a8e-946f-7f9b41630a98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:35:57.726102 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.726056    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81b64d0-ded6-4a8e-946f-7f9b41630a98-kube-api-access-g9s56" (OuterVolumeSpecName: "kube-api-access-g9s56") pod "d81b64d0-ded6-4a8e-946f-7f9b41630a98" (UID: "d81b64d0-ded6-4a8e-946f-7f9b41630a98"). InnerVolumeSpecName "kube-api-access-g9s56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:35:57.825523 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.825490    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-envoy\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:57.825523 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.825518    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-token\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:57.825523 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.825526    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-data\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:57.825769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.825536    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81b64d0-ded6-4a8e-946f-7f9b41630a98-istio-podinfo\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:57.825769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:57.825548    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9s56\" (UniqueName: \"kubernetes.io/projected/d81b64d0-ded6-4a8e-946f-7f9b41630a98-kube-api-access-g9s56\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:35:58.394353 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:58.394314    2565 generic.go:356] "Generic (PLEG): container finished" podID="d81b64d0-ded6-4a8e-946f-7f9b41630a98" containerID="ab2e0e7bbfd393502338ef6a7277ad354c964158fefc534a720719332182f502" exitCode=0
Jul 17 22:35:58.394639 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:58.394358    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd" event={"ID":"d81b64d0-ded6-4a8e-946f-7f9b41630a98","Type":"ContainerDied","Data":"ab2e0e7bbfd393502338ef6a7277ad354c964158fefc534a720719332182f502"}
Jul 17 22:35:58.394639 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:58.394419    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd" event={"ID":"d81b64d0-ded6-4a8e-946f-7f9b41630a98","Type":"ContainerDied","Data":"67affc56ac632477bfbfe886dce573ea3446f1fd54b0996d8202359619cd37c9"}
Jul 17 22:35:58.394639 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:58.394442    2565 scope.go:117] "RemoveContainer" containerID="ab2e0e7bbfd393502338ef6a7277ad354c964158fefc534a720719332182f502"
Jul 17 22:35:58.394639 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:58.394370    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"
Jul 17 22:35:58.404176 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:58.404157    2565 scope.go:117] "RemoveContainer" containerID="d44b3bac6e18412cb9bc748ede7daee7e4ee66ef7708c7fcfdf35e5e3a76751d"
Jul 17 22:35:58.412867 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:58.412846    2565 scope.go:117] "RemoveContainer" containerID="ab2e0e7bbfd393502338ef6a7277ad354c964158fefc534a720719332182f502"
Jul 17 22:35:58.413184 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:35:58.413165    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2e0e7bbfd393502338ef6a7277ad354c964158fefc534a720719332182f502\": container with ID starting with ab2e0e7bbfd393502338ef6a7277ad354c964158fefc534a720719332182f502 not found: ID does not exist" containerID="ab2e0e7bbfd393502338ef6a7277ad354c964158fefc534a720719332182f502"
Jul 17 22:35:58.413247 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:58.413194    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2e0e7bbfd393502338ef6a7277ad354c964158fefc534a720719332182f502"} err="failed to get container status \"ab2e0e7bbfd393502338ef6a7277ad354c964158fefc534a720719332182f502\": rpc error: code = NotFound desc = could not find container \"ab2e0e7bbfd393502338ef6a7277ad354c964158fefc534a720719332182f502\": container with ID starting with ab2e0e7bbfd393502338ef6a7277ad354c964158fefc534a720719332182f502 not found: ID does not exist"
Jul 17 22:35:58.413247 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:58.413213    2565 scope.go:117] "RemoveContainer" containerID="d44b3bac6e18412cb9bc748ede7daee7e4ee66ef7708c7fcfdf35e5e3a76751d"
Jul 17 22:35:58.413469 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:35:58.413453    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44b3bac6e18412cb9bc748ede7daee7e4ee66ef7708c7fcfdf35e5e3a76751d\": container with ID starting with d44b3bac6e18412cb9bc748ede7daee7e4ee66ef7708c7fcfdf35e5e3a76751d not found: ID does not exist" containerID="d44b3bac6e18412cb9bc748ede7daee7e4ee66ef7708c7fcfdf35e5e3a76751d"
Jul 17 22:35:58.413509 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:58.413477    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44b3bac6e18412cb9bc748ede7daee7e4ee66ef7708c7fcfdf35e5e3a76751d"} err="failed to get container status \"d44b3bac6e18412cb9bc748ede7daee7e4ee66ef7708c7fcfdf35e5e3a76751d\": rpc error: code = NotFound desc = could not find container \"d44b3bac6e18412cb9bc748ede7daee7e4ee66ef7708c7fcfdf35e5e3a76751d\": container with ID starting with d44b3bac6e18412cb9bc748ede7daee7e4ee66ef7708c7fcfdf35e5e3a76751d not found: ID does not exist"
Jul 17 22:35:58.418790 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:58.418761    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"]
Jul 17 22:35:58.423308 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:58.423285    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bf8fd3-openshift-default-678f968bc-5zgmd"]
Jul 17 22:35:58.868185 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:58.868150    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:58.968759 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:58.968721    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:35:59.182482 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:35:59.182405    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d81b64d0-ded6-4a8e-946f-7f9b41630a98" path="/var/lib/kubelet/pods/d81b64d0-ded6-4a8e-946f-7f9b41630a98/volumes"
Jul 17 22:36:00.855349 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:00.855317    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:01.241798 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.241706    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:01.378107 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.378051    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:36:01.528337 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.528303    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"]
Jul 17 22:36:01.528844 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.528830    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81b64d0-ded6-4a8e-946f-7f9b41630a98" containerName="istio-proxy"
Jul 17 22:36:01.528888 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.528848    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81b64d0-ded6-4a8e-946f-7f9b41630a98" containerName="istio-proxy"
Jul 17 22:36:01.528960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.528949    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81b64d0-ded6-4a8e-946f-7f9b41630a98" containerName="istio-proxy"
Jul 17 22:36:01.528992 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.528967    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d81b64d0-ded6-4a8e-946f-7f9b41630a98" containerName="istio-proxy"
Jul 17 22:36:01.533868 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.533843    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.537132 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.537108    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-fb05ec-openshift-default-dockercfg-sdgcd\""
Jul 17 22:36:01.546470 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.546441    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"]
Jul 17 22:36:01.565204 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.565169    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:01.663694 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.663650    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df489087-22f5-4d73-878b-1d5deed40d78-istio-podinfo\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.663868 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.663709    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-credential-socket\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.663868 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.663804    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnf6v\" (UniqueName: \"kubernetes.io/projected/df489087-22f5-4d73-878b-1d5deed40d78-kube-api-access-wnf6v\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.663868 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.663859    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-workload-certs\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.663972 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.663925    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-workload-socket\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.663972 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.663958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-istio-envoy\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.664042 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.663975    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-istio-data\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.664042 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.663996    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df489087-22f5-4d73-878b-1d5deed40d78-istiod-ca-cert\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.664123 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.664059    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df489087-22f5-4d73-878b-1d5deed40d78-istio-token\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.716582 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.716546    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:01.765346 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.765286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-workload-socket\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.765346 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.765348    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-istio-envoy\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.765616 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.765374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-istio-data\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.765616 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.765407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df489087-22f5-4d73-878b-1d5deed40d78-istiod-ca-cert\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.765616 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.765466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df489087-22f5-4d73-878b-1d5deed40d78-istio-token\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.765616 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.765497    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df489087-22f5-4d73-878b-1d5deed40d78-istio-podinfo\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.765616 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.765562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-credential-socket\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.765895 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.765621    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnf6v\" (UniqueName: \"kubernetes.io/projected/df489087-22f5-4d73-878b-1d5deed40d78-kube-api-access-wnf6v\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.765895 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.765701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-workload-certs\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.765895 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.765788    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-workload-socket\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.766147 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.766120    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-workload-certs\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.766570 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.766550    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-credential-socket\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.766785 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.766759    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-istio-data\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.767699 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.767506    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df489087-22f5-4d73-878b-1d5deed40d78-istiod-ca-cert\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.769769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.769749    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df489087-22f5-4d73-878b-1d5deed40d78-istio-podinfo\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.769769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.769763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-istio-envoy\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.774841 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.774816    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df489087-22f5-4d73-878b-1d5deed40d78-istio-token\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.775521 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.775503    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnf6v\" (UniqueName: \"kubernetes.io/projected/df489087-22f5-4d73-878b-1d5deed40d78-kube-api-access-wnf6v\") pod \"e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") " pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:01.846194 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:01.846037    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:02.000393 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:02.000365    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"]
Jul 17 22:36:02.002386 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:36:02.002360    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf489087_22f5_4d73_878b_1d5deed40d78.slice/crio-0dec833eebfff8c124da6dfb6c41edab0688342520d9ea9a91bc2b0edf91b315 WatchSource:0}: Error finding container 0dec833eebfff8c124da6dfb6c41edab0688342520d9ea9a91bc2b0edf91b315: Status 404 returned error can't find the container with id 0dec833eebfff8c124da6dfb6c41edab0688342520d9ea9a91bc2b0edf91b315
Jul 17 22:36:02.005039 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:02.005005    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:36:02.005141 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:02.005092    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:36:02.005141 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:02.005125    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:36:02.418139 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:02.418098    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd" event={"ID":"df489087-22f5-4d73-878b-1d5deed40d78","Type":"ContainerStarted","Data":"d5cd36cab17e2923fde35943d9ba1658ccd91c3a146c75c6e87ece6caef15241"}
Jul 17 22:36:02.418324 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:02.418160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd" event={"ID":"df489087-22f5-4d73-878b-1d5deed40d78","Type":"ContainerStarted","Data":"0dec833eebfff8c124da6dfb6c41edab0688342520d9ea9a91bc2b0edf91b315"}
Jul 17 22:36:02.445807 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:02.445752    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd" podStartSLOduration=1.445733879 podStartE2EDuration="1.445733879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:36:02.444304068 +0000 UTC m=+3025.833320719" watchObservedRunningTime="2026-07-17 22:36:02.445733879 +0000 UTC m=+3025.834750521"
Jul 17 22:36:02.847040 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:02.847000    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:03.852398 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:03.852369    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:04.428126 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:04.428084    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:04.429046 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:04.429025    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:04.495298 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:04.495254    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:04.634003 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:04.633974    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:04.743172 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:04.743026    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:04.964647 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:04.964613    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:05.179946 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:05.179902    2565 scope.go:117] "RemoveContainer" containerID="e219ea66f759ca47ccdd92f67cc7fbc5767efe792079085c10bbf0d576d909b9"
Jul 17 22:36:05.432993 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:05.432895    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:05.525815 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:05.525779    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:06.440504 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:06.440415    2565 generic.go:356] "Generic (PLEG): container finished" podID="423dca54-e885-4c31-ac52-805543e0ee5e" containerID="2ffa4fd091b9d1735febb317e3c061064f91c064f2e7851d806850ffed354970" exitCode=6
Jul 17 22:36:06.440504 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:06.440486    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456" event={"ID":"423dca54-e885-4c31-ac52-805543e0ee5e","Type":"ContainerDied","Data":"2ffa4fd091b9d1735febb317e3c061064f91c064f2e7851d806850ffed354970"}
Jul 17 22:36:06.440994 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:06.440544    2565 scope.go:117] "RemoveContainer" containerID="e219ea66f759ca47ccdd92f67cc7fbc5767efe792079085c10bbf0d576d909b9"
Jul 17 22:36:06.440994 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:06.440898    2565 scope.go:117] "RemoveContainer" containerID="2ffa4fd091b9d1735febb317e3c061064f91c064f2e7851d806850ffed354970"
Jul 17 22:36:06.441171 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:36:06.441147    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-cv456_odh-ai-gateway-infra(423dca54-e885-4c31-ac52-805543e0ee5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456" podUID="423dca54-e885-4c31-ac52-805543e0ee5e"
Jul 17 22:36:07.478322 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:07.478277    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456"]
Jul 17 22:36:07.623655 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:07.623620    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456"
Jul 17 22:36:07.628221 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:07.628187    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423dca54-e885-4c31-ac52-805543e0ee5e-maas-api-service-ca\") pod \"423dca54-e885-4c31-ac52-805543e0ee5e\" (UID: \"423dca54-e885-4c31-ac52-805543e0ee5e\") "
Jul 17 22:36:07.628517 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:07.628495    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423dca54-e885-4c31-ac52-805543e0ee5e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "423dca54-e885-4c31-ac52-805543e0ee5e" (UID: "423dca54-e885-4c31-ac52-805543e0ee5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:36:07.628628 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:07.628610    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423dca54-e885-4c31-ac52-805543e0ee5e-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:36:08.455159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:08.455127    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456"
Jul 17 22:36:08.455357 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:08.455126    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456" event={"ID":"423dca54-e885-4c31-ac52-805543e0ee5e","Type":"ContainerDied","Data":"8f3ea29c8a217c0263367a27d2402e0b6b0bdc61c68ed08b303918db526f6d62"}
Jul 17 22:36:08.455357 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:08.455289    2565 scope.go:117] "RemoveContainer" containerID="2ffa4fd091b9d1735febb317e3c061064f91c064f2e7851d806850ffed354970"
Jul 17 22:36:08.479194 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:08.479122    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456"]
Jul 17 22:36:08.481505 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:08.481479    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-cv456"]
Jul 17 22:36:08.882021 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:08.881985    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:09.182891 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:09.182807    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="423dca54-e885-4c31-ac52-805543e0ee5e" path="/var/lib/kubelet/pods/423dca54-e885-4c31-ac52-805543e0ee5e/volumes"
Jul 17 22:36:09.462270 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:09.462176    2565 generic.go:356] "Generic (PLEG): container finished" podID="722b2832-2199-480b-ac12-f84cc1beb2dc" containerID="3fd9fe91b8f0ec045d01a0712bba2f2293edc91e278710af560e1dfb72befdc0" exitCode=0
Jul 17 22:36:09.462270 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:09.462253    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x" event={"ID":"722b2832-2199-480b-ac12-f84cc1beb2dc","Type":"ContainerDied","Data":"3fd9fe91b8f0ec045d01a0712bba2f2293edc91e278710af560e1dfb72befdc0"}
Jul 17 22:36:09.462686 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:09.462672    2565 scope.go:117] "RemoveContainer" containerID="3fd9fe91b8f0ec045d01a0712bba2f2293edc91e278710af560e1dfb72befdc0"
Jul 17 22:36:09.463806 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:09.463778    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:36:09.463879 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:09.463865    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:36:09.463921 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:09.463899    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:36:10.376005 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:10.375943    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:36:10.480306 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:10.480251    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x" event={"ID":"722b2832-2199-480b-ac12-f84cc1beb2dc","Type":"ContainerStarted","Data":"9f9babfa55438f6a05654daedc9fc083db0c7334854761862ab40f223c4511e4"}
Jul 17 22:36:10.687301 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:10.687207    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:36:10.688563 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:10.688535    2565 patch_prober.go:28] interesting pod/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Jul 17 22:36:10.688673 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:10.688592    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x" podUID="722b2832-2199-480b-ac12-f84cc1beb2dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Jul 17 22:36:11.045207 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:11.045166    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:36:11.687622 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:11.687572    2565 patch_prober.go:28] interesting pod/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Jul 17 22:36:11.688160 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:11.687647    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x" podUID="722b2832-2199-480b-ac12-f84cc1beb2dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Jul 17 22:36:12.687289 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:12.687241    2565 patch_prober.go:28] interesting pod/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Jul 17 22:36:12.687509 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:12.687324    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x" podUID="722b2832-2199-480b-ac12-f84cc1beb2dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Jul 17 22:36:13.687429 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:13.687380    2565 patch_prober.go:28] interesting pod/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Jul 17 22:36:13.687973 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:13.687452    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x" podUID="722b2832-2199-480b-ac12-f84cc1beb2dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Jul 17 22:36:14.690703 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:14.690673    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:36:14.691154 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:14.690944    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:36:14.691655 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:14.691635    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:36:16.472934 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:16.472902    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:17.902693 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:17.902656    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:18.003978 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:18.003938    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:19.430232 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:19.430198    2565 patch_prober.go:28] interesting pod/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Jul 17 22:36:19.430730 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:19.430258    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd" podUID="df489087-22f5-4d73-878b-1d5deed40d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Jul 17 22:36:19.537139 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:19.537099    2565 generic.go:356] "Generic (PLEG): container finished" podID="df489087-22f5-4d73-878b-1d5deed40d78" containerID="d5cd36cab17e2923fde35943d9ba1658ccd91c3a146c75c6e87ece6caef15241" exitCode=0
Jul 17 22:36:19.537515 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:19.537258    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd" event={"ID":"df489087-22f5-4d73-878b-1d5deed40d78","Type":"ContainerDied","Data":"d5cd36cab17e2923fde35943d9ba1658ccd91c3a146c75c6e87ece6caef15241"}
Jul 17 22:36:19.538161 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:19.538139    2565 scope.go:117] "RemoveContainer" containerID="d5cd36cab17e2923fde35943d9ba1658ccd91c3a146c75c6e87ece6caef15241"
Jul 17 22:36:19.539589 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:19.539557    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:36:19.539686 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:19.539623    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:36:19.539686 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:19.539654    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 22:36:20.167883 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:20.167845    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:36:20.545355 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:20.545321    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd" event={"ID":"df489087-22f5-4d73-878b-1d5deed40d78","Type":"ContainerStarted","Data":"4176535d4c81fdd935bab80cf982eff4bf412f9892f82d702145b1c04eb6d253"}
Jul 17 22:36:20.846465 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:20.846356    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:20.846685 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:20.846658    2565 patch_prober.go:28] interesting pod/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Jul 17 22:36:20.846766 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:20.846732    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd" podUID="df489087-22f5-4d73-878b-1d5deed40d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Jul 17 22:36:21.846723 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:21.846679    2565 patch_prober.go:28] interesting pod/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Jul 17 22:36:21.847180 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:21.846748    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd" podUID="df489087-22f5-4d73-878b-1d5deed40d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Jul 17 22:36:22.846430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:22.846397    2565 patch_prober.go:28] interesting pod/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Jul 17 22:36:22.846614 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:22.846451    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd" podUID="df489087-22f5-4d73-878b-1d5deed40d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Jul 17 22:36:23.847156 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:23.847114    2565 patch_prober.go:28] interesting pod/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Jul 17 22:36:23.847583 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:23.847182    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd" podUID="df489087-22f5-4d73-878b-1d5deed40d78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Jul 17 22:36:24.850447 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:24.850416    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:24.850820 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:24.850695    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:24.851513 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:24.851498    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:36:26.711615 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.711577    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t"]
Jul 17 22:36:26.712034 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.711973    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423dca54-e885-4c31-ac52-805543e0ee5e" containerName="revoke-keys"
Jul 17 22:36:26.712034 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.711985    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="423dca54-e885-4c31-ac52-805543e0ee5e" containerName="revoke-keys"
Jul 17 22:36:26.712034 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.711999    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81b64d0-ded6-4a8e-946f-7f9b41630a98" containerName="istio-proxy"
Jul 17 22:36:26.712034 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.712005    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81b64d0-ded6-4a8e-946f-7f9b41630a98" containerName="istio-proxy"
Jul 17 22:36:26.712034 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.712016    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423dca54-e885-4c31-ac52-805543e0ee5e" containerName="revoke-keys"
Jul 17 22:36:26.712034 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.712021    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="423dca54-e885-4c31-ac52-805543e0ee5e" containerName="revoke-keys"
Jul 17 22:36:26.712261 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.712106    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="423dca54-e885-4c31-ac52-805543e0ee5e" containerName="revoke-keys"
Jul 17 22:36:26.712261 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.712118    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="423dca54-e885-4c31-ac52-805543e0ee5e" containerName="revoke-keys"
Jul 17 22:36:26.712261 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.712131    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="423dca54-e885-4c31-ac52-805543e0ee5e" containerName="revoke-keys"
Jul 17 22:36:26.716850 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.716830    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t"
Jul 17 22:36:26.721103 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.721082    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:36:26.722679 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.722654    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:36:26.725387 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.725363    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t"]
Jul 17 22:36:26.829131 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.829061    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/674b6ae1-1d22-4332-bbd4-4e213f3e8180-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t\" (UID: \"674b6ae1-1d22-4332-bbd4-4e213f3e8180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t"
Jul 17 22:36:26.930228 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.930188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/674b6ae1-1d22-4332-bbd4-4e213f3e8180-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t\" (UID: \"674b6ae1-1d22-4332-bbd4-4e213f3e8180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t"
Jul 17 22:36:26.930869 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:26.930850    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/674b6ae1-1d22-4332-bbd4-4e213f3e8180-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t\" (UID: \"674b6ae1-1d22-4332-bbd4-4e213f3e8180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t"
Jul 17 22:36:27.028690 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:27.028652    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t"
Jul 17 22:36:27.174248 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:27.174219    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t"]
Jul 17 22:36:27.176419 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:36:27.176375    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674b6ae1_1d22_4332_bbd4_4e213f3e8180.slice/crio-eda3e7724ec8b8061f635da9187fd27f4a9fe75e402c4a2a63bcbd8427dee740 WatchSource:0}: Error finding container eda3e7724ec8b8061f635da9187fd27f4a9fe75e402c4a2a63bcbd8427dee740: Status 404 returned error can't find the container with id eda3e7724ec8b8061f635da9187fd27f4a9fe75e402c4a2a63bcbd8427dee740
Jul 17 22:36:27.584551 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:27.584518    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t" event={"ID":"674b6ae1-1d22-4332-bbd4-4e213f3e8180","Type":"ContainerStarted","Data":"eecb28b2ea6d30835b9968b97b72defa70d75c81bac711008ce3c3c7b4af739d"}
Jul 17 22:36:27.584743 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:27.584570    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t" event={"ID":"674b6ae1-1d22-4332-bbd4-4e213f3e8180","Type":"ContainerStarted","Data":"eda3e7724ec8b8061f635da9187fd27f4a9fe75e402c4a2a63bcbd8427dee740"}
Jul 17 22:36:27.711802 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:27.711715    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:27.899035 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:27.898989    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:28.590644 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:28.590609    2565 generic.go:356] "Generic (PLEG): container finished" podID="674b6ae1-1d22-4332-bbd4-4e213f3e8180" containerID="eecb28b2ea6d30835b9968b97b72defa70d75c81bac711008ce3c3c7b4af739d" exitCode=6
Jul 17 22:36:28.590835 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:28.590696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t" event={"ID":"674b6ae1-1d22-4332-bbd4-4e213f3e8180","Type":"ContainerDied","Data":"eecb28b2ea6d30835b9968b97b72defa70d75c81bac711008ce3c3c7b4af739d"}
Jul 17 22:36:28.590885 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:28.590870    2565 scope.go:117] "RemoveContainer" containerID="eecb28b2ea6d30835b9968b97b72defa70d75c81bac711008ce3c3c7b4af739d"
Jul 17 22:36:29.078019 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:29.077961    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:29.598404 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:29.598301    2565 generic.go:356] "Generic (PLEG): container finished" podID="674b6ae1-1d22-4332-bbd4-4e213f3e8180" containerID="934750302c319628638140fa438d694b047ef340057bb77e767faab813830582" exitCode=6
Jul 17 22:36:29.598404 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:29.598375    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t" event={"ID":"674b6ae1-1d22-4332-bbd4-4e213f3e8180","Type":"ContainerDied","Data":"934750302c319628638140fa438d694b047ef340057bb77e767faab813830582"}
Jul 17 22:36:29.598740 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:29.598434    2565 scope.go:117] "RemoveContainer" containerID="eecb28b2ea6d30835b9968b97b72defa70d75c81bac711008ce3c3c7b4af739d"
Jul 17 22:36:29.598740 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:29.598685    2565 scope.go:117] "RemoveContainer" containerID="934750302c319628638140fa438d694b047ef340057bb77e767faab813830582"
Jul 17 22:36:29.598938 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:36:29.598914    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t_odh-ai-gateway-infra(674b6ae1-1d22-4332-bbd4-4e213f3e8180)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t" podUID="674b6ae1-1d22-4332-bbd4-4e213f3e8180"
Jul 17 22:36:30.605440 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:30.605405    2565 scope.go:117] "RemoveContainer" containerID="934750302c319628638140fa438d694b047ef340057bb77e767faab813830582"
Jul 17 22:36:30.605887 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:36:30.605685    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t_odh-ai-gateway-infra(674b6ae1-1d22-4332-bbd4-4e213f3e8180)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t" podUID="674b6ae1-1d22-4332-bbd4-4e213f3e8180"
Jul 17 22:36:37.692424 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:37.692389    2565 scope.go:117] "RemoveContainer" containerID="ee626e7a27406b2de8de632a8df3bd832d930cd64ea15f36634b1e4862d6410f"
Jul 17 22:36:41.404523 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:41.404490    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:36:44.754078 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:44.754036    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:36:45.176709 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:45.176609    2565 scope.go:117] "RemoveContainer" containerID="934750302c319628638140fa438d694b047ef340057bb77e767faab813830582"
Jul 17 22:36:45.237711 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:45.237677    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb"]
Jul 17 22:36:45.238083 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:45.238049    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423dca54-e885-4c31-ac52-805543e0ee5e" containerName="revoke-keys"
Jul 17 22:36:45.238083 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:45.238079    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="423dca54-e885-4c31-ac52-805543e0ee5e" containerName="revoke-keys"
Jul 17 22:36:45.242741 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:45.242722    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb"
Jul 17 22:36:45.258687 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:45.258659    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb"]
Jul 17 22:36:45.301791 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:45.301748    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb84400-6443-4d47-a88d-402444055a07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-spdzb\" (UID: \"2bb84400-6443-4d47-a88d-402444055a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb"
Jul 17 22:36:45.402406 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:45.402372    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb84400-6443-4d47-a88d-402444055a07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-spdzb\" (UID: \"2bb84400-6443-4d47-a88d-402444055a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb"
Jul 17 22:36:45.403016 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:45.402996    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb84400-6443-4d47-a88d-402444055a07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-spdzb\" (UID: \"2bb84400-6443-4d47-a88d-402444055a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb"
Jul 17 22:36:45.552975 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:45.552932    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb"
Jul 17 22:36:45.682809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:45.682774    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t" event={"ID":"674b6ae1-1d22-4332-bbd4-4e213f3e8180","Type":"ContainerStarted","Data":"67b90a572eb48ce15b6f0f3861a51932bd1416fc3375bca42b82b89779f231fc"}
Jul 17 22:36:45.698398 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:45.698358    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb"]
Jul 17 22:36:45.702032 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:36:45.701993    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb84400_6443_4d47_a88d_402444055a07.slice/crio-95a84cb9caec5c0a57a124f365903702121bea78bd3d924f434fca6fd4e2a40a WatchSource:0}: Error finding container 95a84cb9caec5c0a57a124f365903702121bea78bd3d924f434fca6fd4e2a40a: Status 404 returned error can't find the container with id 95a84cb9caec5c0a57a124f365903702121bea78bd3d924f434fca6fd4e2a40a
Jul 17 22:36:45.702431 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:45.702384    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t" podStartSLOduration=19.702365449 podStartE2EDuration="19.702365449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:36:27.621749838 +0000 UTC m=+3051.010766489" watchObservedRunningTime="2026-07-17 22:36:45.702365449 +0000 UTC m=+3069.091382100"
Jul 17 22:36:46.689143 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:46.689036    2565 generic.go:356] "Generic (PLEG): container finished" podID="2bb84400-6443-4d47-a88d-402444055a07" containerID="a20078a6ae4b38b660941b3913d3df4f3289d9e382e263cb0356095c955265f4" exitCode=6
Jul 17 22:36:46.689143 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:46.689120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb" event={"ID":"2bb84400-6443-4d47-a88d-402444055a07","Type":"ContainerDied","Data":"a20078a6ae4b38b660941b3913d3df4f3289d9e382e263cb0356095c955265f4"}
Jul 17 22:36:46.689632 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:46.689175    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb" event={"ID":"2bb84400-6443-4d47-a88d-402444055a07","Type":"ContainerStarted","Data":"95a84cb9caec5c0a57a124f365903702121bea78bd3d924f434fca6fd4e2a40a"}
Jul 17 22:36:46.689632 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:46.689330    2565 scope.go:117] "RemoveContainer" containerID="a20078a6ae4b38b660941b3913d3df4f3289d9e382e263cb0356095c955265f4"
Jul 17 22:36:46.691027 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:46.691005    2565 generic.go:356] "Generic (PLEG): container finished" podID="674b6ae1-1d22-4332-bbd4-4e213f3e8180" containerID="67b90a572eb48ce15b6f0f3861a51932bd1416fc3375bca42b82b89779f231fc" exitCode=6
Jul 17 22:36:46.691167 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:46.691103    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t" event={"ID":"674b6ae1-1d22-4332-bbd4-4e213f3e8180","Type":"ContainerDied","Data":"67b90a572eb48ce15b6f0f3861a51932bd1416fc3375bca42b82b89779f231fc"}
Jul 17 22:36:46.691167 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:46.691149    2565 scope.go:117] "RemoveContainer" containerID="934750302c319628638140fa438d694b047ef340057bb77e767faab813830582"
Jul 17 22:36:46.691415 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:46.691395    2565 scope.go:117] "RemoveContainer" containerID="67b90a572eb48ce15b6f0f3861a51932bd1416fc3375bca42b82b89779f231fc"
Jul 17 22:36:46.691600 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:36:46.691581    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t_odh-ai-gateway-infra(674b6ae1-1d22-4332-bbd4-4e213f3e8180)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t" podUID="674b6ae1-1d22-4332-bbd4-4e213f3e8180"
Jul 17 22:36:46.718191 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:46.718156    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t"]
Jul 17 22:36:47.698549 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:47.698437    2565 generic.go:356] "Generic (PLEG): container finished" podID="2bb84400-6443-4d47-a88d-402444055a07" containerID="5dfdabbf8aee8a58578e701610ecf6b14791203008e868d751ef3555f24f5968" exitCode=6
Jul 17 22:36:47.698549 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:47.698515    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb" event={"ID":"2bb84400-6443-4d47-a88d-402444055a07","Type":"ContainerDied","Data":"5dfdabbf8aee8a58578e701610ecf6b14791203008e868d751ef3555f24f5968"}
Jul 17 22:36:47.699045 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:47.698587    2565 scope.go:117] "RemoveContainer" containerID="a20078a6ae4b38b660941b3913d3df4f3289d9e382e263cb0356095c955265f4"
Jul 17 22:36:47.699045 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:47.698830    2565 scope.go:117] "RemoveContainer" containerID="5dfdabbf8aee8a58578e701610ecf6b14791203008e868d751ef3555f24f5968"
Jul 17 22:36:47.699183 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:36:47.699060    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-spdzb_odh-ai-gateway-infra(2bb84400-6443-4d47-a88d-402444055a07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb" podUID="2bb84400-6443-4d47-a88d-402444055a07"
Jul 17 22:36:47.841602 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:47.841575    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t"
Jul 17 22:36:47.922969 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:47.922941    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/674b6ae1-1d22-4332-bbd4-4e213f3e8180-maas-api-service-ca\") pod \"674b6ae1-1d22-4332-bbd4-4e213f3e8180\" (UID: \"674b6ae1-1d22-4332-bbd4-4e213f3e8180\") "
Jul 17 22:36:47.923347 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:47.923324    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674b6ae1-1d22-4332-bbd4-4e213f3e8180-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "674b6ae1-1d22-4332-bbd4-4e213f3e8180" (UID: "674b6ae1-1d22-4332-bbd4-4e213f3e8180"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:36:48.024341 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.024292    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/674b6ae1-1d22-4332-bbd4-4e213f3e8180-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:36:48.497020 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.496986    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8"]
Jul 17 22:36:48.497424 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.497410    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="674b6ae1-1d22-4332-bbd4-4e213f3e8180" containerName="revoke-keys"
Jul 17 22:36:48.497484 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.497425    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="674b6ae1-1d22-4332-bbd4-4e213f3e8180" containerName="revoke-keys"
Jul 17 22:36:48.497484 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.497434    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="674b6ae1-1d22-4332-bbd4-4e213f3e8180" containerName="revoke-keys"
Jul 17 22:36:48.497484 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.497440    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="674b6ae1-1d22-4332-bbd4-4e213f3e8180" containerName="revoke-keys"
Jul 17 22:36:48.497484 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.497450    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="674b6ae1-1d22-4332-bbd4-4e213f3e8180" containerName="revoke-keys"
Jul 17 22:36:48.497484 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.497455    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="674b6ae1-1d22-4332-bbd4-4e213f3e8180" containerName="revoke-keys"
Jul 17 22:36:48.497642 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.497504    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="674b6ae1-1d22-4332-bbd4-4e213f3e8180" containerName="revoke-keys"
Jul 17 22:36:48.497642 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.497512    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="674b6ae1-1d22-4332-bbd4-4e213f3e8180" containerName="revoke-keys"
Jul 17 22:36:48.500613 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.500596    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8"
Jul 17 22:36:48.511550 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.511512    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8"]
Jul 17 22:36:48.528438 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.528394    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619484e1-31d6-4904-a8a6-44dc250c4148-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8\" (UID: \"619484e1-31d6-4904-a8a6-44dc250c4148\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8"
Jul 17 22:36:48.630025 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.629983    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619484e1-31d6-4904-a8a6-44dc250c4148-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8\" (UID: \"619484e1-31d6-4904-a8a6-44dc250c4148\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8"
Jul 17 22:36:48.630657 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.630634    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619484e1-31d6-4904-a8a6-44dc250c4148-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8\" (UID: \"619484e1-31d6-4904-a8a6-44dc250c4148\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8"
Jul 17 22:36:48.707044 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.707011    2565 scope.go:117] "RemoveContainer" containerID="5dfdabbf8aee8a58578e701610ecf6b14791203008e868d751ef3555f24f5968"
Jul 17 22:36:48.707552 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:36:48.707346    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-spdzb_odh-ai-gateway-infra(2bb84400-6443-4d47-a88d-402444055a07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb" podUID="2bb84400-6443-4d47-a88d-402444055a07"
Jul 17 22:36:48.708134 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.708097    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t" event={"ID":"674b6ae1-1d22-4332-bbd4-4e213f3e8180","Type":"ContainerDied","Data":"eda3e7724ec8b8061f635da9187fd27f4a9fe75e402c4a2a63bcbd8427dee740"}
Jul 17 22:36:48.708134 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.708123    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t"
Jul 17 22:36:48.708340 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.708154    2565 scope.go:117] "RemoveContainer" containerID="67b90a572eb48ce15b6f0f3861a51932bd1416fc3375bca42b82b89779f231fc"
Jul 17 22:36:48.735784 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.735747    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t"]
Jul 17 22:36:48.741230 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.741201    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-lbt2t"]
Jul 17 22:36:48.821340 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.821241    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8"
Jul 17 22:36:48.977319 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:48.977291    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8"]
Jul 17 22:36:48.979143 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:36:48.979107    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619484e1_31d6_4904_a8a6_44dc250c4148.slice/crio-3922b8d364e07c2b68670c4f242a8c07d9759b92a9330eddcdc5927281b905d4 WatchSource:0}: Error finding container 3922b8d364e07c2b68670c4f242a8c07d9759b92a9330eddcdc5927281b905d4: Status 404 returned error can't find the container with id 3922b8d364e07c2b68670c4f242a8c07d9759b92a9330eddcdc5927281b905d4
Jul 17 22:36:49.182095 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:49.182032    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="674b6ae1-1d22-4332-bbd4-4e213f3e8180" path="/var/lib/kubelet/pods/674b6ae1-1d22-4332-bbd4-4e213f3e8180/volumes"
Jul 17 22:36:49.716029 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:49.715999    2565 generic.go:356] "Generic (PLEG): container finished" podID="619484e1-31d6-4904-a8a6-44dc250c4148" containerID="7cf136ae9e064fd54bebee0b3a1b64ef0a33a9bc204825efa71f53678a3187f8" exitCode=6
Jul 17 22:36:49.716505 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:49.716095    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8" event={"ID":"619484e1-31d6-4904-a8a6-44dc250c4148","Type":"ContainerDied","Data":"7cf136ae9e064fd54bebee0b3a1b64ef0a33a9bc204825efa71f53678a3187f8"}
Jul 17 22:36:49.716505 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:49.716149    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8" event={"ID":"619484e1-31d6-4904-a8a6-44dc250c4148","Type":"ContainerStarted","Data":"3922b8d364e07c2b68670c4f242a8c07d9759b92a9330eddcdc5927281b905d4"}
Jul 17 22:36:49.716505 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:49.716274    2565 scope.go:117] "RemoveContainer" containerID="7cf136ae9e064fd54bebee0b3a1b64ef0a33a9bc204825efa71f53678a3187f8"
Jul 17 22:36:50.725077 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:50.724969    2565 generic.go:356] "Generic (PLEG): container finished" podID="619484e1-31d6-4904-a8a6-44dc250c4148" containerID="dc004eeb8ca36472dfa30944325265e3f415ff7e95cafa120565a3f039fc6676" exitCode=6
Jul 17 22:36:50.725077 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:50.725022    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8" event={"ID":"619484e1-31d6-4904-a8a6-44dc250c4148","Type":"ContainerDied","Data":"dc004eeb8ca36472dfa30944325265e3f415ff7e95cafa120565a3f039fc6676"}
Jul 17 22:36:50.725598 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:50.725119    2565 scope.go:117] "RemoveContainer" containerID="7cf136ae9e064fd54bebee0b3a1b64ef0a33a9bc204825efa71f53678a3187f8"
Jul 17 22:36:50.725598 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:50.725266    2565 scope.go:117] "RemoveContainer" containerID="dc004eeb8ca36472dfa30944325265e3f415ff7e95cafa120565a3f039fc6676"
Jul 17 22:36:50.725598 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:36:50.725519    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8_odh-ai-gateway-infra(619484e1-31d6-4904-a8a6-44dc250c4148)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8" podUID="619484e1-31d6-4904-a8a6-44dc250c4148"
Jul 17 22:36:51.732237 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:36:51.732205    2565 scope.go:117] "RemoveContainer" containerID="dc004eeb8ca36472dfa30944325265e3f415ff7e95cafa120565a3f039fc6676"
Jul 17 22:36:51.732624 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:36:51.732432    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8_odh-ai-gateway-infra(619484e1-31d6-4904-a8a6-44dc250c4148)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8" podUID="619484e1-31d6-4904-a8a6-44dc250c4148"
Jul 17 22:37:00.956502 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:00.956459    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:37:01.177163 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:01.177130    2565 scope.go:117] "RemoveContainer" containerID="5dfdabbf8aee8a58578e701610ecf6b14791203008e868d751ef3555f24f5968"
Jul 17 22:37:01.789925 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:01.789893    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb" event={"ID":"2bb84400-6443-4d47-a88d-402444055a07","Type":"ContainerStarted","Data":"4e43f1be0cb2b57327c60fdc98e9ab2e7daeb6daa2d7f43a45c26e0712db9ae0"}
Jul 17 22:37:01.812599 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:01.812547    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb" podStartSLOduration=16.812530236 podStartE2EDuration="16.812530236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:37:01.812418853 +0000 UTC m=+3085.201435524" watchObservedRunningTime="2026-07-17 22:37:01.812530236 +0000 UTC m=+3085.201546886"
Jul 17 22:37:02.799624 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:02.799581    2565 generic.go:356] "Generic (PLEG): container finished" podID="2bb84400-6443-4d47-a88d-402444055a07" containerID="4e43f1be0cb2b57327c60fdc98e9ab2e7daeb6daa2d7f43a45c26e0712db9ae0" exitCode=6
Jul 17 22:37:02.800133 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:02.799661    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb" event={"ID":"2bb84400-6443-4d47-a88d-402444055a07","Type":"ContainerDied","Data":"4e43f1be0cb2b57327c60fdc98e9ab2e7daeb6daa2d7f43a45c26e0712db9ae0"}
Jul 17 22:37:02.800133 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:02.799754    2565 scope.go:117] "RemoveContainer" containerID="5dfdabbf8aee8a58578e701610ecf6b14791203008e868d751ef3555f24f5968"
Jul 17 22:37:02.800133 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:02.800060    2565 scope.go:117] "RemoveContainer" containerID="4e43f1be0cb2b57327c60fdc98e9ab2e7daeb6daa2d7f43a45c26e0712db9ae0"
Jul 17 22:37:02.800382 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:37:02.800360    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-spdzb_odh-ai-gateway-infra(2bb84400-6443-4d47-a88d-402444055a07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb" podUID="2bb84400-6443-4d47-a88d-402444055a07"
Jul 17 22:37:02.821884 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:02.821851    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb"]
Jul 17 22:37:03.114237 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:03.114143    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf"]
Jul 17 22:37:03.114632 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:03.114617    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="674b6ae1-1d22-4332-bbd4-4e213f3e8180" containerName="revoke-keys"
Jul 17 22:37:03.119012 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:03.118992    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf"
Jul 17 22:37:03.125712 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:03.125680    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf"]
Jul 17 22:37:03.172436 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:03.172395    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2f181c-9535-4bc8-8ecb-9a44a15e2ead-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf\" (UID: \"4b2f181c-9535-4bc8-8ecb-9a44a15e2ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf"
Jul 17 22:37:03.273469 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:03.273425    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2f181c-9535-4bc8-8ecb-9a44a15e2ead-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf\" (UID: \"4b2f181c-9535-4bc8-8ecb-9a44a15e2ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf"
Jul 17 22:37:03.274169 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:03.274149    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2f181c-9535-4bc8-8ecb-9a44a15e2ead-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf\" (UID: \"4b2f181c-9535-4bc8-8ecb-9a44a15e2ead\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf"
Jul 17 22:37:03.432564 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:03.432460    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf"
Jul 17 22:37:03.572434 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:03.572408    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf"]
Jul 17 22:37:03.575166 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:37:03.575135    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2f181c_9535_4bc8_8ecb_9a44a15e2ead.slice/crio-b41b4e163895316e0cb3091b760833d774e925bde95886f1663fd2cd84a01bdd WatchSource:0}: Error finding container b41b4e163895316e0cb3091b760833d774e925bde95886f1663fd2cd84a01bdd: Status 404 returned error can't find the container with id b41b4e163895316e0cb3091b760833d774e925bde95886f1663fd2cd84a01bdd
Jul 17 22:37:03.807555 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:03.807527    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf" event={"ID":"4b2f181c-9535-4bc8-8ecb-9a44a15e2ead","Type":"ContainerStarted","Data":"b41b4e163895316e0cb3091b760833d774e925bde95886f1663fd2cd84a01bdd"}
Jul 17 22:37:03.938552 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:03.938524    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb"
Jul 17 22:37:03.979918 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:03.979882    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb84400-6443-4d47-a88d-402444055a07-maas-api-service-ca\") pod \"2bb84400-6443-4d47-a88d-402444055a07\" (UID: \"2bb84400-6443-4d47-a88d-402444055a07\") "
Jul 17 22:37:03.980302 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:03.980277    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb84400-6443-4d47-a88d-402444055a07-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2bb84400-6443-4d47-a88d-402444055a07" (UID: "2bb84400-6443-4d47-a88d-402444055a07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:37:04.080661 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:04.080576    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb84400-6443-4d47-a88d-402444055a07-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:37:04.815132 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:04.815096    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb" event={"ID":"2bb84400-6443-4d47-a88d-402444055a07","Type":"ContainerDied","Data":"95a84cb9caec5c0a57a124f365903702121bea78bd3d924f434fca6fd4e2a40a"}
Jul 17 22:37:04.815584 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:04.815173    2565 scope.go:117] "RemoveContainer" containerID="4e43f1be0cb2b57327c60fdc98e9ab2e7daeb6daa2d7f43a45c26e0712db9ae0"
Jul 17 22:37:04.815584 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:04.815105    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb"
Jul 17 22:37:04.816942 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:04.816915    2565 generic.go:356] "Generic (PLEG): container finished" podID="4b2f181c-9535-4bc8-8ecb-9a44a15e2ead" containerID="c8a0389eb892031eb2550b5dcf26e3c3db635ca16812d0436eabfee512044f1f" exitCode=6
Jul 17 22:37:04.817129 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:04.816967    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf" event={"ID":"4b2f181c-9535-4bc8-8ecb-9a44a15e2ead","Type":"ContainerDied","Data":"c8a0389eb892031eb2550b5dcf26e3c3db635ca16812d0436eabfee512044f1f"}
Jul 17 22:37:04.817280 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:04.817237    2565 scope.go:117] "RemoveContainer" containerID="c8a0389eb892031eb2550b5dcf26e3c3db635ca16812d0436eabfee512044f1f"
Jul 17 22:37:04.858014 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:04.857982    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb"]
Jul 17 22:37:04.864494 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:04.864465    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-spdzb"]
Jul 17 22:37:05.177220 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:05.177189    2565 scope.go:117] "RemoveContainer" containerID="dc004eeb8ca36472dfa30944325265e3f415ff7e95cafa120565a3f039fc6676"
Jul 17 22:37:05.183431 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:05.183404    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bb84400-6443-4d47-a88d-402444055a07" path="/var/lib/kubelet/pods/2bb84400-6443-4d47-a88d-402444055a07/volumes"
Jul 17 22:37:05.824217 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:05.824178    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8" event={"ID":"619484e1-31d6-4904-a8a6-44dc250c4148","Type":"ContainerStarted","Data":"b54f0980089604735f79507f3f1fb5b2ec59e31dd0767b5e256e6a6f982242f2"}
Jul 17 22:37:05.826833 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:05.826805    2565 generic.go:356] "Generic (PLEG): container finished" podID="4b2f181c-9535-4bc8-8ecb-9a44a15e2ead" containerID="c25d04fc1d30808e4b169230b0eb2caf6f533f3b9fa2931a099ad665f63b7feb" exitCode=6
Jul 17 22:37:05.827003 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:05.826858    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf" event={"ID":"4b2f181c-9535-4bc8-8ecb-9a44a15e2ead","Type":"ContainerDied","Data":"c25d04fc1d30808e4b169230b0eb2caf6f533f3b9fa2931a099ad665f63b7feb"}
Jul 17 22:37:05.827003 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:05.826901    2565 scope.go:117] "RemoveContainer" containerID="c8a0389eb892031eb2550b5dcf26e3c3db635ca16812d0436eabfee512044f1f"
Jul 17 22:37:05.827190 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:05.827168    2565 scope.go:117] "RemoveContainer" containerID="c25d04fc1d30808e4b169230b0eb2caf6f533f3b9fa2931a099ad665f63b7feb"
Jul 17 22:37:05.827426 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:37:05.827404    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf_odh-ai-gateway-infra(4b2f181c-9535-4bc8-8ecb-9a44a15e2ead)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf" podUID="4b2f181c-9535-4bc8-8ecb-9a44a15e2ead"
Jul 17 22:37:05.840403 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:05.840352    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8" podStartSLOduration=17.840333231 podStartE2EDuration="17.840333231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:37:05.839695138 +0000 UTC m=+3089.228711801" watchObservedRunningTime="2026-07-17 22:37:05.840333231 +0000 UTC m=+3089.229349882"
Jul 17 22:37:06.833424 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:06.833395    2565 scope.go:117] "RemoveContainer" containerID="c25d04fc1d30808e4b169230b0eb2caf6f533f3b9fa2931a099ad665f63b7feb"
Jul 17 22:37:06.833876 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:37:06.833653    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf_odh-ai-gateway-infra(4b2f181c-9535-4bc8-8ecb-9a44a15e2ead)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf" podUID="4b2f181c-9535-4bc8-8ecb-9a44a15e2ead"
Jul 17 22:37:06.834978 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:06.834952    2565 generic.go:356] "Generic (PLEG): container finished" podID="619484e1-31d6-4904-a8a6-44dc250c4148" containerID="b54f0980089604735f79507f3f1fb5b2ec59e31dd0767b5e256e6a6f982242f2" exitCode=6
Jul 17 22:37:06.835106 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:06.835009    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8" event={"ID":"619484e1-31d6-4904-a8a6-44dc250c4148","Type":"ContainerDied","Data":"b54f0980089604735f79507f3f1fb5b2ec59e31dd0767b5e256e6a6f982242f2"}
Jul 17 22:37:06.835106 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:06.835054    2565 scope.go:117] "RemoveContainer" containerID="dc004eeb8ca36472dfa30944325265e3f415ff7e95cafa120565a3f039fc6676"
Jul 17 22:37:06.835330 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:06.835311    2565 scope.go:117] "RemoveContainer" containerID="b54f0980089604735f79507f3f1fb5b2ec59e31dd0767b5e256e6a6f982242f2"
Jul 17 22:37:06.835569 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:37:06.835548    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8_odh-ai-gateway-infra(619484e1-31d6-4904-a8a6-44dc250c4148)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8" podUID="619484e1-31d6-4904-a8a6-44dc250c4148"
Jul 17 22:37:06.849374 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:06.849341    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8"]
Jul 17 22:37:07.985613 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:07.985587    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8"
Jul 17 22:37:08.121651 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:08.121550    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619484e1-31d6-4904-a8a6-44dc250c4148-maas-api-service-ca\") pod \"619484e1-31d6-4904-a8a6-44dc250c4148\" (UID: \"619484e1-31d6-4904-a8a6-44dc250c4148\") "
Jul 17 22:37:08.121906 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:08.121882    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619484e1-31d6-4904-a8a6-44dc250c4148-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "619484e1-31d6-4904-a8a6-44dc250c4148" (UID: "619484e1-31d6-4904-a8a6-44dc250c4148"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:37:08.223021 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:08.222976    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619484e1-31d6-4904-a8a6-44dc250c4148-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:37:08.848092 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:08.848036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8" event={"ID":"619484e1-31d6-4904-a8a6-44dc250c4148","Type":"ContainerDied","Data":"3922b8d364e07c2b68670c4f242a8c07d9759b92a9330eddcdc5927281b905d4"}
Jul 17 22:37:08.848092 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:08.848093    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8"
Jul 17 22:37:08.848336 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:08.848126    2565 scope.go:117] "RemoveContainer" containerID="b54f0980089604735f79507f3f1fb5b2ec59e31dd0767b5e256e6a6f982242f2"
Jul 17 22:37:08.876020 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:08.875991    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8"]
Jul 17 22:37:08.881555 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:08.881525    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8jxj8"]
Jul 17 22:37:09.182752 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:09.182673    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="619484e1-31d6-4904-a8a6-44dc250c4148" path="/var/lib/kubelet/pods/619484e1-31d6-4904-a8a6-44dc250c4148/volumes"
Jul 17 22:37:17.546849 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:17.546814    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:37:19.176806 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:19.176763    2565 scope.go:117] "RemoveContainer" containerID="c25d04fc1d30808e4b169230b0eb2caf6f533f3b9fa2931a099ad665f63b7feb"
Jul 17 22:37:19.906779 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:19.906746    2565 generic.go:356] "Generic (PLEG): container finished" podID="4b2f181c-9535-4bc8-8ecb-9a44a15e2ead" containerID="e438d68405aff7e105c823c75405301efbc54a45eae9ea2c6ce7e82bc9b3288e" exitCode=6
Jul 17 22:37:19.906953 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:19.906803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf" event={"ID":"4b2f181c-9535-4bc8-8ecb-9a44a15e2ead","Type":"ContainerDied","Data":"e438d68405aff7e105c823c75405301efbc54a45eae9ea2c6ce7e82bc9b3288e"}
Jul 17 22:37:19.906953 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:19.906860    2565 scope.go:117] "RemoveContainer" containerID="c25d04fc1d30808e4b169230b0eb2caf6f533f3b9fa2931a099ad665f63b7feb"
Jul 17 22:37:19.907286 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:19.907265    2565 scope.go:117] "RemoveContainer" containerID="e438d68405aff7e105c823c75405301efbc54a45eae9ea2c6ce7e82bc9b3288e"
Jul 17 22:37:19.907533 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:37:19.907514    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf_odh-ai-gateway-infra(4b2f181c-9535-4bc8-8ecb-9a44a15e2ead)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf" podUID="4b2f181c-9535-4bc8-8ecb-9a44a15e2ead"
Jul 17 22:37:20.937594 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:20.937552    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf"]
Jul 17 22:37:21.095378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:21.095352    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf"
Jul 17 22:37:21.249815 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:21.249789    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2f181c-9535-4bc8-8ecb-9a44a15e2ead-maas-api-service-ca\") pod \"4b2f181c-9535-4bc8-8ecb-9a44a15e2ead\" (UID: \"4b2f181c-9535-4bc8-8ecb-9a44a15e2ead\") "
Jul 17 22:37:21.250242 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:21.250219    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2f181c-9535-4bc8-8ecb-9a44a15e2ead-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4b2f181c-9535-4bc8-8ecb-9a44a15e2ead" (UID: "4b2f181c-9535-4bc8-8ecb-9a44a15e2ead"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:37:21.351286 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:21.351243    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b2f181c-9535-4bc8-8ecb-9a44a15e2ead-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:37:21.920352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:21.920324    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf"
Jul 17 22:37:21.920352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:21.920345    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf" event={"ID":"4b2f181c-9535-4bc8-8ecb-9a44a15e2ead","Type":"ContainerDied","Data":"b41b4e163895316e0cb3091b760833d774e925bde95886f1663fd2cd84a01bdd"}
Jul 17 22:37:21.920602 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:21.920409    2565 scope.go:117] "RemoveContainer" containerID="e438d68405aff7e105c823c75405301efbc54a45eae9ea2c6ce7e82bc9b3288e"
Jul 17 22:37:21.942438 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:21.942408    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf"]
Jul 17 22:37:21.948589 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:21.948566    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8hfxf"]
Jul 17 22:37:23.183266 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:23.183229    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b2f181c-9535-4bc8-8ecb-9a44a15e2ead" path="/var/lib/kubelet/pods/4b2f181c-9535-4bc8-8ecb-9a44a15e2ead/volumes"
Jul 17 22:37:26.757151 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757113    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g"]
Jul 17 22:37:26.757535 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757489    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bb84400-6443-4d47-a88d-402444055a07" containerName="revoke-keys"
Jul 17 22:37:26.757535 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757500    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bb84400-6443-4d47-a88d-402444055a07" containerName="revoke-keys"
Jul 17 22:37:26.757535 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757513    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bb84400-6443-4d47-a88d-402444055a07" containerName="revoke-keys"
Jul 17 22:37:26.757535 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757519    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bb84400-6443-4d47-a88d-402444055a07" containerName="revoke-keys"
Jul 17 22:37:26.757535 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757527    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bb84400-6443-4d47-a88d-402444055a07" containerName="revoke-keys"
Jul 17 22:37:26.757535 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757533    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bb84400-6443-4d47-a88d-402444055a07" containerName="revoke-keys"
Jul 17 22:37:26.757535 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757539    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619484e1-31d6-4904-a8a6-44dc250c4148" containerName="revoke-keys"
Jul 17 22:37:26.757755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757545    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="619484e1-31d6-4904-a8a6-44dc250c4148" containerName="revoke-keys"
Jul 17 22:37:26.757755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757555    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619484e1-31d6-4904-a8a6-44dc250c4148" containerName="revoke-keys"
Jul 17 22:37:26.757755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757560    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="619484e1-31d6-4904-a8a6-44dc250c4148" containerName="revoke-keys"
Jul 17 22:37:26.757755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757566    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2f181c-9535-4bc8-8ecb-9a44a15e2ead" containerName="revoke-keys"
Jul 17 22:37:26.757755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757571    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2f181c-9535-4bc8-8ecb-9a44a15e2ead" containerName="revoke-keys"
Jul 17 22:37:26.757755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757579    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2f181c-9535-4bc8-8ecb-9a44a15e2ead" containerName="revoke-keys"
Jul 17 22:37:26.757755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757583    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2f181c-9535-4bc8-8ecb-9a44a15e2ead" containerName="revoke-keys"
Jul 17 22:37:26.757755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757636    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="619484e1-31d6-4904-a8a6-44dc250c4148" containerName="revoke-keys"
Jul 17 22:37:26.757755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757642    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="619484e1-31d6-4904-a8a6-44dc250c4148" containerName="revoke-keys"
Jul 17 22:37:26.757755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757648    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bb84400-6443-4d47-a88d-402444055a07" containerName="revoke-keys"
Jul 17 22:37:26.757755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757653    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b2f181c-9535-4bc8-8ecb-9a44a15e2ead" containerName="revoke-keys"
Jul 17 22:37:26.757755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757659    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b2f181c-9535-4bc8-8ecb-9a44a15e2ead" containerName="revoke-keys"
Jul 17 22:37:26.757755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757667    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bb84400-6443-4d47-a88d-402444055a07" containerName="revoke-keys"
Jul 17 22:37:26.757755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.757677    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bb84400-6443-4d47-a88d-402444055a07" containerName="revoke-keys"
Jul 17 22:37:26.762299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.762282    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g"
Jul 17 22:37:26.765177 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.765151    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:37:26.765795 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.765772    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:37:26.767715 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.767688    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g"]
Jul 17 22:37:26.899583 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:26.899542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f1a233-130f-4cc6-8df6-fd58f73327e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g\" (UID: \"c7f1a233-130f-4cc6-8df6-fd58f73327e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g"
Jul 17 22:37:27.000298 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:27.000260    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f1a233-130f-4cc6-8df6-fd58f73327e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g\" (UID: \"c7f1a233-130f-4cc6-8df6-fd58f73327e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g"
Jul 17 22:37:27.000934 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:27.000905    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f1a233-130f-4cc6-8df6-fd58f73327e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g\" (UID: \"c7f1a233-130f-4cc6-8df6-fd58f73327e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g"
Jul 17 22:37:27.074494 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:27.074396    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g"
Jul 17 22:37:27.214135 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:27.214107    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g"]
Jul 17 22:37:27.216099 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:37:27.216053    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f1a233_130f_4cc6_8df6_fd58f73327e2.slice/crio-f9b796c7b6667a7290c8e2f4eb4b69dd47e4ba96f0fafa46ebc0850d9b0e8922 WatchSource:0}: Error finding container f9b796c7b6667a7290c8e2f4eb4b69dd47e4ba96f0fafa46ebc0850d9b0e8922: Status 404 returned error can't find the container with id f9b796c7b6667a7290c8e2f4eb4b69dd47e4ba96f0fafa46ebc0850d9b0e8922
Jul 17 22:37:27.951949 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:27.951913    2565 generic.go:356] "Generic (PLEG): container finished" podID="c7f1a233-130f-4cc6-8df6-fd58f73327e2" containerID="2f431ccbd0761ccecfb614b7733475c821c7706838440fc6c760db71267a6a7e" exitCode=6
Jul 17 22:37:27.952387 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:27.951994    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g" event={"ID":"c7f1a233-130f-4cc6-8df6-fd58f73327e2","Type":"ContainerDied","Data":"2f431ccbd0761ccecfb614b7733475c821c7706838440fc6c760db71267a6a7e"}
Jul 17 22:37:27.952387 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:27.952055    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g" event={"ID":"c7f1a233-130f-4cc6-8df6-fd58f73327e2","Type":"ContainerStarted","Data":"f9b796c7b6667a7290c8e2f4eb4b69dd47e4ba96f0fafa46ebc0850d9b0e8922"}
Jul 17 22:37:27.952387 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:27.952202    2565 scope.go:117] "RemoveContainer" containerID="2f431ccbd0761ccecfb614b7733475c821c7706838440fc6c760db71267a6a7e"
Jul 17 22:37:28.959222 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:28.959131    2565 generic.go:356] "Generic (PLEG): container finished" podID="c7f1a233-130f-4cc6-8df6-fd58f73327e2" containerID="a17cf53e6095c89f20aa37822d426f51b4256f383d642f560c0ed0b07c4ef2e3" exitCode=6
Jul 17 22:37:28.959656 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:28.959217    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g" event={"ID":"c7f1a233-130f-4cc6-8df6-fd58f73327e2","Type":"ContainerDied","Data":"a17cf53e6095c89f20aa37822d426f51b4256f383d642f560c0ed0b07c4ef2e3"}
Jul 17 22:37:28.959656 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:28.959283    2565 scope.go:117] "RemoveContainer" containerID="2f431ccbd0761ccecfb614b7733475c821c7706838440fc6c760db71267a6a7e"
Jul 17 22:37:28.959656 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:28.959495    2565 scope.go:117] "RemoveContainer" containerID="a17cf53e6095c89f20aa37822d426f51b4256f383d642f560c0ed0b07c4ef2e3"
Jul 17 22:37:28.959773 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:37:28.959745    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g_odh-ai-gateway-infra(c7f1a233-130f-4cc6-8df6-fd58f73327e2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g" podUID="c7f1a233-130f-4cc6-8df6-fd58f73327e2"
Jul 17 22:37:29.965770 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:29.965743    2565 scope.go:117] "RemoveContainer" containerID="a17cf53e6095c89f20aa37822d426f51b4256f383d642f560c0ed0b07c4ef2e3"
Jul 17 22:37:29.966190 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:37:29.965957    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g_odh-ai-gateway-infra(c7f1a233-130f-4cc6-8df6-fd58f73327e2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g" podUID="c7f1a233-130f-4cc6-8df6-fd58f73327e2"
Jul 17 22:37:32.559379 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:32.559344    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:37:35.282590 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:35.282554    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr"]
Jul 17 22:37:35.282981 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:35.282904    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2f181c-9535-4bc8-8ecb-9a44a15e2ead" containerName="revoke-keys"
Jul 17 22:37:35.282981 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:35.282914    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2f181c-9535-4bc8-8ecb-9a44a15e2ead" containerName="revoke-keys"
Jul 17 22:37:35.282981 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:35.282942    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619484e1-31d6-4904-a8a6-44dc250c4148" containerName="revoke-keys"
Jul 17 22:37:35.282981 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:35.282948    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="619484e1-31d6-4904-a8a6-44dc250c4148" containerName="revoke-keys"
Jul 17 22:37:35.283157 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:35.283002    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="619484e1-31d6-4904-a8a6-44dc250c4148" containerName="revoke-keys"
Jul 17 22:37:35.283157 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:35.283011    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b2f181c-9535-4bc8-8ecb-9a44a15e2ead" containerName="revoke-keys"
Jul 17 22:37:35.287124 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:35.287105    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr"
Jul 17 22:37:35.304185 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:35.304156    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr"]
Jul 17 22:37:35.376827 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:35.376791    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0edce941-b48a-4428-96be-73c500f726d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr\" (UID: \"0edce941-b48a-4428-96be-73c500f726d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr"
Jul 17 22:37:35.478391 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:35.478345    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0edce941-b48a-4428-96be-73c500f726d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr\" (UID: \"0edce941-b48a-4428-96be-73c500f726d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr"
Jul 17 22:37:35.478981 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:35.478960    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0edce941-b48a-4428-96be-73c500f726d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr\" (UID: \"0edce941-b48a-4428-96be-73c500f726d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr"
Jul 17 22:37:35.597162 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:35.597052    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr"
Jul 17 22:37:35.739655 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:35.739632    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr"]
Jul 17 22:37:35.741798 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:37:35.741772    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0edce941_b48a_4428_96be_73c500f726d9.slice/crio-be76e3d9456dc95fb69fa68c9db061bac4d79dbcb2cb1512cb18f44e6153e738 WatchSource:0}: Error finding container be76e3d9456dc95fb69fa68c9db061bac4d79dbcb2cb1512cb18f44e6153e738: Status 404 returned error can't find the container with id be76e3d9456dc95fb69fa68c9db061bac4d79dbcb2cb1512cb18f44e6153e738
Jul 17 22:37:35.997693 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:35.997662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr" event={"ID":"0edce941-b48a-4428-96be-73c500f726d9","Type":"ContainerStarted","Data":"be76e3d9456dc95fb69fa68c9db061bac4d79dbcb2cb1512cb18f44e6153e738"}
Jul 17 22:37:37.003632 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:37.003601    2565 generic.go:356] "Generic (PLEG): container finished" podID="0edce941-b48a-4428-96be-73c500f726d9" containerID="9b932a1dfcd4086613fae113850fec5b0d0aad711c2de2b2dc516f98382667c9" exitCode=6
Jul 17 22:37:37.004035 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:37.003682    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr" event={"ID":"0edce941-b48a-4428-96be-73c500f726d9","Type":"ContainerDied","Data":"9b932a1dfcd4086613fae113850fec5b0d0aad711c2de2b2dc516f98382667c9"}
Jul 17 22:37:37.004035 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:37.003854    2565 scope.go:117] "RemoveContainer" containerID="9b932a1dfcd4086613fae113850fec5b0d0aad711c2de2b2dc516f98382667c9"
Jul 17 22:37:38.010662 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:38.010624    2565 generic.go:356] "Generic (PLEG): container finished" podID="0edce941-b48a-4428-96be-73c500f726d9" containerID="5ef42011d3b1fb979693a0d4aeb6460a0c5a28431b5c57bfaed6f44575a8fcd8" exitCode=6
Jul 17 22:37:38.011146 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:38.010700    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr" event={"ID":"0edce941-b48a-4428-96be-73c500f726d9","Type":"ContainerDied","Data":"5ef42011d3b1fb979693a0d4aeb6460a0c5a28431b5c57bfaed6f44575a8fcd8"}
Jul 17 22:37:38.011146 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:38.010754    2565 scope.go:117] "RemoveContainer" containerID="9b932a1dfcd4086613fae113850fec5b0d0aad711c2de2b2dc516f98382667c9"
Jul 17 22:37:38.011146 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:38.010932    2565 scope.go:117] "RemoveContainer" containerID="5ef42011d3b1fb979693a0d4aeb6460a0c5a28431b5c57bfaed6f44575a8fcd8"
Jul 17 22:37:38.011335 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:37:38.011232    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr_odh-ai-gateway-infra(0edce941-b48a-4428-96be-73c500f726d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr" podUID="0edce941-b48a-4428-96be-73c500f726d9"
Jul 17 22:37:39.018324 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:39.018293    2565 scope.go:117] "RemoveContainer" containerID="5ef42011d3b1fb979693a0d4aeb6460a0c5a28431b5c57bfaed6f44575a8fcd8"
Jul 17 22:37:39.018736 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:37:39.018503    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr_odh-ai-gateway-infra(0edce941-b48a-4428-96be-73c500f726d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr" podUID="0edce941-b48a-4428-96be-73c500f726d9"
Jul 17 22:37:45.176652 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:45.176611    2565 scope.go:117] "RemoveContainer" containerID="a17cf53e6095c89f20aa37822d426f51b4256f383d642f560c0ed0b07c4ef2e3"
Jul 17 22:37:46.058117 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:46.058061    2565 generic.go:356] "Generic (PLEG): container finished" podID="c7f1a233-130f-4cc6-8df6-fd58f73327e2" containerID="6582e8e19601f3ddc433712c2dd0592cc77f6cab9d69ce252dda0f7f53cd9ac4" exitCode=6
Jul 17 22:37:46.058263 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:46.058128    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g" event={"ID":"c7f1a233-130f-4cc6-8df6-fd58f73327e2","Type":"ContainerDied","Data":"6582e8e19601f3ddc433712c2dd0592cc77f6cab9d69ce252dda0f7f53cd9ac4"}
Jul 17 22:37:46.058263 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:46.058189    2565 scope.go:117] "RemoveContainer" containerID="a17cf53e6095c89f20aa37822d426f51b4256f383d642f560c0ed0b07c4ef2e3"
Jul 17 22:37:46.058573 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:46.058557    2565 scope.go:117] "RemoveContainer" containerID="6582e8e19601f3ddc433712c2dd0592cc77f6cab9d69ce252dda0f7f53cd9ac4"
Jul 17 22:37:46.058793 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:37:46.058775    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g_odh-ai-gateway-infra(c7f1a233-130f-4cc6-8df6-fd58f73327e2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g" podUID="c7f1a233-130f-4cc6-8df6-fd58f73327e2"
Jul 17 22:37:47.095360 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:47.095325    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g"]
Jul 17 22:37:47.238704 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:47.238679    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g"
Jul 17 22:37:47.293641 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:47.293606    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f1a233-130f-4cc6-8df6-fd58f73327e2-maas-api-service-ca\") pod \"c7f1a233-130f-4cc6-8df6-fd58f73327e2\" (UID: \"c7f1a233-130f-4cc6-8df6-fd58f73327e2\") "
Jul 17 22:37:47.293965 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:47.293942    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f1a233-130f-4cc6-8df6-fd58f73327e2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c7f1a233-130f-4cc6-8df6-fd58f73327e2" (UID: "c7f1a233-130f-4cc6-8df6-fd58f73327e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:37:47.394897 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:47.394811    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f1a233-130f-4cc6-8df6-fd58f73327e2-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:37:48.070590 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.070558    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g"
Jul 17 22:37:48.070782 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.070607    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g" event={"ID":"c7f1a233-130f-4cc6-8df6-fd58f73327e2","Type":"ContainerDied","Data":"f9b796c7b6667a7290c8e2f4eb4b69dd47e4ba96f0fafa46ebc0850d9b0e8922"}
Jul 17 22:37:48.070782 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.070665    2565 scope.go:117] "RemoveContainer" containerID="6582e8e19601f3ddc433712c2dd0592cc77f6cab9d69ce252dda0f7f53cd9ac4"
Jul 17 22:37:48.105908 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.105877    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g"]
Jul 17 22:37:48.116282 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.116249    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-pkw2g"]
Jul 17 22:37:48.547881 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.547835    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg"]
Jul 17 22:37:48.548255 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.548241    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f1a233-130f-4cc6-8df6-fd58f73327e2" containerName="revoke-keys"
Jul 17 22:37:48.548255 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.548256    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f1a233-130f-4cc6-8df6-fd58f73327e2" containerName="revoke-keys"
Jul 17 22:37:48.548348 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.548264    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f1a233-130f-4cc6-8df6-fd58f73327e2" containerName="revoke-keys"
Jul 17 22:37:48.548348 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.548269    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f1a233-130f-4cc6-8df6-fd58f73327e2" containerName="revoke-keys"
Jul 17 22:37:48.548348 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.548279    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f1a233-130f-4cc6-8df6-fd58f73327e2" containerName="revoke-keys"
Jul 17 22:37:48.548348 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.548285    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f1a233-130f-4cc6-8df6-fd58f73327e2" containerName="revoke-keys"
Jul 17 22:37:48.548348 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.548336    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f1a233-130f-4cc6-8df6-fd58f73327e2" containerName="revoke-keys"
Jul 17 22:37:48.548348 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.548344    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f1a233-130f-4cc6-8df6-fd58f73327e2" containerName="revoke-keys"
Jul 17 22:37:48.548348 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.548350    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f1a233-130f-4cc6-8df6-fd58f73327e2" containerName="revoke-keys"
Jul 17 22:37:48.552542 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.552523    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg"
Jul 17 22:37:48.560853 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.560825    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg"]
Jul 17 22:37:48.605385 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.605347    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d68724b-39ba-404c-83ee-1703da3583ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg\" (UID: \"5d68724b-39ba-404c-83ee-1703da3583ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg"
Jul 17 22:37:48.706164 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.706128    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d68724b-39ba-404c-83ee-1703da3583ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg\" (UID: \"5d68724b-39ba-404c-83ee-1703da3583ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg"
Jul 17 22:37:48.706768 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.706750    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d68724b-39ba-404c-83ee-1703da3583ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg\" (UID: \"5d68724b-39ba-404c-83ee-1703da3583ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg"
Jul 17 22:37:48.864761 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:48.864660    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg"
Jul 17 22:37:49.159845 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:49.159735    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:37:49.182801 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:49.182761    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7f1a233-130f-4cc6-8df6-fd58f73327e2" path="/var/lib/kubelet/pods/c7f1a233-130f-4cc6-8df6-fd58f73327e2/volumes"
Jul 17 22:37:49.221491 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:49.221455    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg"]
Jul 17 22:37:49.227658 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:37:49.227621    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d68724b_39ba_404c_83ee_1703da3583ec.slice/crio-a955d1b2a83fd285dfd1990cb01b132aac99d537c67584c06e4937bb762445ca WatchSource:0}: Error finding container a955d1b2a83fd285dfd1990cb01b132aac99d537c67584c06e4937bb762445ca: Status 404 returned error can't find the container with id a955d1b2a83fd285dfd1990cb01b132aac99d537c67584c06e4937bb762445ca
Jul 17 22:37:50.084759 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:50.084726    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d68724b-39ba-404c-83ee-1703da3583ec" containerID="3f7d1a452e00daada36239cf91353416c9a0a9ccb3ab8b184c4767ae624a57c4" exitCode=6
Jul 17 22:37:50.084951 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:50.084780    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg" event={"ID":"5d68724b-39ba-404c-83ee-1703da3583ec","Type":"ContainerDied","Data":"3f7d1a452e00daada36239cf91353416c9a0a9ccb3ab8b184c4767ae624a57c4"}
Jul 17 22:37:50.084951 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:50.084817    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg" event={"ID":"5d68724b-39ba-404c-83ee-1703da3583ec","Type":"ContainerStarted","Data":"a955d1b2a83fd285dfd1990cb01b132aac99d537c67584c06e4937bb762445ca"}
Jul 17 22:37:50.085051 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:50.084999    2565 scope.go:117] "RemoveContainer" containerID="3f7d1a452e00daada36239cf91353416c9a0a9ccb3ab8b184c4767ae624a57c4"
Jul 17 22:37:50.177225 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:50.177191    2565 scope.go:117] "RemoveContainer" containerID="5ef42011d3b1fb979693a0d4aeb6460a0c5a28431b5c57bfaed6f44575a8fcd8"
Jul 17 22:37:51.092170 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:51.092137    2565 generic.go:356] "Generic (PLEG): container finished" podID="0edce941-b48a-4428-96be-73c500f726d9" containerID="791522738dce5652a4a7f089a85f4d143eb10ec1135db10d59e6e82fce434a4b" exitCode=6
Jul 17 22:37:51.092366 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:51.092206    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr" event={"ID":"0edce941-b48a-4428-96be-73c500f726d9","Type":"ContainerDied","Data":"791522738dce5652a4a7f089a85f4d143eb10ec1135db10d59e6e82fce434a4b"}
Jul 17 22:37:51.092366 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:51.092260    2565 scope.go:117] "RemoveContainer" containerID="5ef42011d3b1fb979693a0d4aeb6460a0c5a28431b5c57bfaed6f44575a8fcd8"
Jul 17 22:37:51.092694 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:51.092673    2565 scope.go:117] "RemoveContainer" containerID="791522738dce5652a4a7f089a85f4d143eb10ec1135db10d59e6e82fce434a4b"
Jul 17 22:37:51.092911 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:37:51.092893    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr_odh-ai-gateway-infra(0edce941-b48a-4428-96be-73c500f726d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr" podUID="0edce941-b48a-4428-96be-73c500f726d9"
Jul 17 22:37:51.094317 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:51.094295    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d68724b-39ba-404c-83ee-1703da3583ec" containerID="e69405b4a6a6ed1e7eb32bac7c8bd84238b6fd91de7580c1a66f53c7191c3d7f" exitCode=6
Jul 17 22:37:51.094425 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:51.094378    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg" event={"ID":"5d68724b-39ba-404c-83ee-1703da3583ec","Type":"ContainerDied","Data":"e69405b4a6a6ed1e7eb32bac7c8bd84238b6fd91de7580c1a66f53c7191c3d7f"}
Jul 17 22:37:51.094593 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:51.094580    2565 scope.go:117] "RemoveContainer" containerID="e69405b4a6a6ed1e7eb32bac7c8bd84238b6fd91de7580c1a66f53c7191c3d7f"
Jul 17 22:37:51.094769 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:37:51.094753    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg_odh-ai-gateway-infra(5d68724b-39ba-404c-83ee-1703da3583ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg" podUID="5d68724b-39ba-404c-83ee-1703da3583ec"
Jul 17 22:37:51.103995 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:51.103971    2565 scope.go:117] "RemoveContainer" containerID="3f7d1a452e00daada36239cf91353416c9a0a9ccb3ab8b184c4767ae624a57c4"
Jul 17 22:37:52.102559 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:52.102534    2565 scope.go:117] "RemoveContainer" containerID="e69405b4a6a6ed1e7eb32bac7c8bd84238b6fd91de7580c1a66f53c7191c3d7f"
Jul 17 22:37:52.103039 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:37:52.102755    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg_odh-ai-gateway-infra(5d68724b-39ba-404c-83ee-1703da3583ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg" podUID="5d68724b-39ba-404c-83ee-1703da3583ec"
Jul 17 22:37:52.137051 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:52.137015    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr"]
Jul 17 22:37:52.277894 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:52.277868    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr"
Jul 17 22:37:52.343293 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:52.343263    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0edce941-b48a-4428-96be-73c500f726d9-maas-api-service-ca\") pod \"0edce941-b48a-4428-96be-73c500f726d9\" (UID: \"0edce941-b48a-4428-96be-73c500f726d9\") "
Jul 17 22:37:52.343676 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:52.343655    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0edce941-b48a-4428-96be-73c500f726d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0edce941-b48a-4428-96be-73c500f726d9" (UID: "0edce941-b48a-4428-96be-73c500f726d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:37:52.444741 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:52.444649    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0edce941-b48a-4428-96be-73c500f726d9-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:37:53.108678 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.108647    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr"
Jul 17 22:37:53.109157 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.108682    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr" event={"ID":"0edce941-b48a-4428-96be-73c500f726d9","Type":"ContainerDied","Data":"be76e3d9456dc95fb69fa68c9db061bac4d79dbcb2cb1512cb18f44e6153e738"}
Jul 17 22:37:53.109157 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.108749    2565 scope.go:117] "RemoveContainer" containerID="791522738dce5652a4a7f089a85f4d143eb10ec1135db10d59e6e82fce434a4b"
Jul 17 22:37:53.131370 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.131341    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr"]
Jul 17 22:37:53.139155 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.139129    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8dmtr"]
Jul 17 22:37:53.159962 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.159936    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2"]
Jul 17 22:37:53.160340 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.160326    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0edce941-b48a-4428-96be-73c500f726d9" containerName="revoke-keys"
Jul 17 22:37:53.160401 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.160343    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0edce941-b48a-4428-96be-73c500f726d9" containerName="revoke-keys"
Jul 17 22:37:53.160401 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.160360    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0edce941-b48a-4428-96be-73c500f726d9" containerName="revoke-keys"
Jul 17 22:37:53.160401 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.160366    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0edce941-b48a-4428-96be-73c500f726d9" containerName="revoke-keys"
Jul 17 22:37:53.160510 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.160428    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0edce941-b48a-4428-96be-73c500f726d9" containerName="revoke-keys"
Jul 17 22:37:53.160510 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.160437    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0edce941-b48a-4428-96be-73c500f726d9" containerName="revoke-keys"
Jul 17 22:37:53.164929 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.164912    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2"
Jul 17 22:37:53.179133 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.175736    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2"]
Jul 17 22:37:53.187507 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.187467    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0edce941-b48a-4428-96be-73c500f726d9" path="/var/lib/kubelet/pods/0edce941-b48a-4428-96be-73c500f726d9/volumes"
Jul 17 22:37:53.252844 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.252798    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b2778f-99a2-47a6-822b-4020f4e78c5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2\" (UID: \"47b2778f-99a2-47a6-822b-4020f4e78c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2"
Jul 17 22:37:53.354303 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.354262    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b2778f-99a2-47a6-822b-4020f4e78c5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2\" (UID: \"47b2778f-99a2-47a6-822b-4020f4e78c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2"
Jul 17 22:37:53.354861 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.354844    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b2778f-99a2-47a6-822b-4020f4e78c5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2\" (UID: \"47b2778f-99a2-47a6-822b-4020f4e78c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2"
Jul 17 22:37:53.480017 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.479931    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2"
Jul 17 22:37:53.620378 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:53.620338    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2"]
Jul 17 22:37:53.623479 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:37:53.623450    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b2778f_99a2_47a6_822b_4020f4e78c5b.slice/crio-2ed66e8a8960ffd422f5f226918611e1c1f385d2f5b75e704a594f241989b4c5 WatchSource:0}: Error finding container 2ed66e8a8960ffd422f5f226918611e1c1f385d2f5b75e704a594f241989b4c5: Status 404 returned error can't find the container with id 2ed66e8a8960ffd422f5f226918611e1c1f385d2f5b75e704a594f241989b4c5
Jul 17 22:37:54.116699 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:54.116668    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2" event={"ID":"47b2778f-99a2-47a6-822b-4020f4e78c5b","Type":"ContainerStarted","Data":"0ed909d188fbefdf739bd35066f62cf30f03c15aec39f2d60e6d32c24c76f19d"}
Jul 17 22:37:54.117106 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:54.116716    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2" event={"ID":"47b2778f-99a2-47a6-822b-4020f4e78c5b","Type":"ContainerStarted","Data":"2ed66e8a8960ffd422f5f226918611e1c1f385d2f5b75e704a594f241989b4c5"}
Jul 17 22:37:54.139346 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:54.139243    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2" podStartSLOduration=1.139226167 podStartE2EDuration="1.139226167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:37:54.13640807 +0000 UTC m=+3137.525424721" watchObservedRunningTime="2026-07-17 22:37:54.139226167 +0000 UTC m=+3137.528242818"
Jul 17 22:37:55.122852 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:55.122813    2565 generic.go:356] "Generic (PLEG): container finished" podID="47b2778f-99a2-47a6-822b-4020f4e78c5b" containerID="0ed909d188fbefdf739bd35066f62cf30f03c15aec39f2d60e6d32c24c76f19d" exitCode=6
Jul 17 22:37:55.123303 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:55.122904    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2" event={"ID":"47b2778f-99a2-47a6-822b-4020f4e78c5b","Type":"ContainerDied","Data":"0ed909d188fbefdf739bd35066f62cf30f03c15aec39f2d60e6d32c24c76f19d"}
Jul 17 22:37:55.123303 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:55.123110    2565 scope.go:117] "RemoveContainer" containerID="0ed909d188fbefdf739bd35066f62cf30f03c15aec39f2d60e6d32c24c76f19d"
Jul 17 22:37:56.129850 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:56.129754    2565 generic.go:356] "Generic (PLEG): container finished" podID="47b2778f-99a2-47a6-822b-4020f4e78c5b" containerID="050615aed7286774b88892f891fdaac3fb2dfdb6336198bc2d58438e3ac3dbde" exitCode=6
Jul 17 22:37:56.129850 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:56.129807    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2" event={"ID":"47b2778f-99a2-47a6-822b-4020f4e78c5b","Type":"ContainerDied","Data":"050615aed7286774b88892f891fdaac3fb2dfdb6336198bc2d58438e3ac3dbde"}
Jul 17 22:37:56.130369 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:56.129853    2565 scope.go:117] "RemoveContainer" containerID="0ed909d188fbefdf739bd35066f62cf30f03c15aec39f2d60e6d32c24c76f19d"
Jul 17 22:37:56.130369 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:56.130082    2565 scope.go:117] "RemoveContainer" containerID="050615aed7286774b88892f891fdaac3fb2dfdb6336198bc2d58438e3ac3dbde"
Jul 17 22:37:56.130369 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:37:56.130341    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2_odh-ai-gateway-infra(47b2778f-99a2-47a6-822b-4020f4e78c5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2" podUID="47b2778f-99a2-47a6-822b-4020f4e78c5b"
Jul 17 22:37:57.136722 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:37:57.136695    2565 scope.go:117] "RemoveContainer" containerID="050615aed7286774b88892f891fdaac3fb2dfdb6336198bc2d58438e3ac3dbde"
Jul 17 22:37:57.137153 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:37:57.136894    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2_odh-ai-gateway-infra(47b2778f-99a2-47a6-822b-4020f4e78c5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2" podUID="47b2778f-99a2-47a6-822b-4020f4e78c5b"
Jul 17 22:38:05.176780 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:05.176703    2565 scope.go:117] "RemoveContainer" containerID="e69405b4a6a6ed1e7eb32bac7c8bd84238b6fd91de7580c1a66f53c7191c3d7f"
Jul 17 22:38:06.184823 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:06.184735    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d68724b-39ba-404c-83ee-1703da3583ec" containerID="54a7442e1b7928d5af326eb8655cc92e018edd23e6af5877be750deb0adcb2e2" exitCode=6
Jul 17 22:38:06.184823 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:06.184803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg" event={"ID":"5d68724b-39ba-404c-83ee-1703da3583ec","Type":"ContainerDied","Data":"54a7442e1b7928d5af326eb8655cc92e018edd23e6af5877be750deb0adcb2e2"}
Jul 17 22:38:06.185301 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:06.184866    2565 scope.go:117] "RemoveContainer" containerID="e69405b4a6a6ed1e7eb32bac7c8bd84238b6fd91de7580c1a66f53c7191c3d7f"
Jul 17 22:38:06.185301 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:06.185219    2565 scope.go:117] "RemoveContainer" containerID="54a7442e1b7928d5af326eb8655cc92e018edd23e6af5877be750deb0adcb2e2"
Jul 17 22:38:06.185447 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:38:06.185423    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg_odh-ai-gateway-infra(5d68724b-39ba-404c-83ee-1703da3583ec)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg" podUID="5d68724b-39ba-404c-83ee-1703da3583ec"
Jul 17 22:38:07.216025 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:07.215989    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg"]
Jul 17 22:38:07.361120 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:07.361094    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg"
Jul 17 22:38:07.497168 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:07.497139    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d68724b-39ba-404c-83ee-1703da3583ec-maas-api-service-ca\") pod \"5d68724b-39ba-404c-83ee-1703da3583ec\" (UID: \"5d68724b-39ba-404c-83ee-1703da3583ec\") "
Jul 17 22:38:07.497508 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:07.497481    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d68724b-39ba-404c-83ee-1703da3583ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d68724b-39ba-404c-83ee-1703da3583ec" (UID: "5d68724b-39ba-404c-83ee-1703da3583ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:38:07.598931 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:07.598891    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d68724b-39ba-404c-83ee-1703da3583ec-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:38:08.197233 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:08.197202    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg"
Jul 17 22:38:08.197233 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:08.197222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg" event={"ID":"5d68724b-39ba-404c-83ee-1703da3583ec","Type":"ContainerDied","Data":"a955d1b2a83fd285dfd1990cb01b132aac99d537c67584c06e4937bb762445ca"}
Jul 17 22:38:08.197489 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:08.197279    2565 scope.go:117] "RemoveContainer" containerID="54a7442e1b7928d5af326eb8655cc92e018edd23e6af5877be750deb0adcb2e2"
Jul 17 22:38:08.218937 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:08.218904    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg"]
Jul 17 22:38:08.223309 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:08.223282    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-lqfwg"]
Jul 17 22:38:09.181964 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:09.181927    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d68724b-39ba-404c-83ee-1703da3583ec" path="/var/lib/kubelet/pods/5d68724b-39ba-404c-83ee-1703da3583ec/volumes"
Jul 17 22:38:11.946258 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:11.946221    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:38:12.176665 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:12.176631    2565 scope.go:117] "RemoveContainer" containerID="050615aed7286774b88892f891fdaac3fb2dfdb6336198bc2d58438e3ac3dbde"
Jul 17 22:38:13.228630 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:13.228542    2565 generic.go:356] "Generic (PLEG): container finished" podID="47b2778f-99a2-47a6-822b-4020f4e78c5b" containerID="35d0daefd583fc1541246505dc43a84dd62b25fc4bfea2da25024387b80385e7" exitCode=6
Jul 17 22:38:13.228630 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:13.228598    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2" event={"ID":"47b2778f-99a2-47a6-822b-4020f4e78c5b","Type":"ContainerDied","Data":"35d0daefd583fc1541246505dc43a84dd62b25fc4bfea2da25024387b80385e7"}
Jul 17 22:38:13.229084 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:13.228647    2565 scope.go:117] "RemoveContainer" containerID="050615aed7286774b88892f891fdaac3fb2dfdb6336198bc2d58438e3ac3dbde"
Jul 17 22:38:13.229084 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:13.228969    2565 scope.go:117] "RemoveContainer" containerID="35d0daefd583fc1541246505dc43a84dd62b25fc4bfea2da25024387b80385e7"
Jul 17 22:38:13.229271 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:38:13.229249    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2_odh-ai-gateway-infra(47b2778f-99a2-47a6-822b-4020f4e78c5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2" podUID="47b2778f-99a2-47a6-822b-4020f4e78c5b"
Jul 17 22:38:14.257528 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:14.257495    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2"]
Jul 17 22:38:14.403118 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:14.403088    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2"
Jul 17 22:38:14.458029 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:14.457978    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b2778f-99a2-47a6-822b-4020f4e78c5b-maas-api-service-ca\") pod \"47b2778f-99a2-47a6-822b-4020f4e78c5b\" (UID: \"47b2778f-99a2-47a6-822b-4020f4e78c5b\") "
Jul 17 22:38:14.458412 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:14.458388    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b2778f-99a2-47a6-822b-4020f4e78c5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47b2778f-99a2-47a6-822b-4020f4e78c5b" (UID: "47b2778f-99a2-47a6-822b-4020f4e78c5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:38:14.559719 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:14.559620    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b2778f-99a2-47a6-822b-4020f4e78c5b-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:38:15.242294 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:15.242261    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2" event={"ID":"47b2778f-99a2-47a6-822b-4020f4e78c5b","Type":"ContainerDied","Data":"2ed66e8a8960ffd422f5f226918611e1c1f385d2f5b75e704a594f241989b4c5"}
Jul 17 22:38:15.242294 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:15.242281    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2"
Jul 17 22:38:15.242569 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:15.242326    2565 scope.go:117] "RemoveContainer" containerID="35d0daefd583fc1541246505dc43a84dd62b25fc4bfea2da25024387b80385e7"
Jul 17 22:38:15.258909 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:15.258877    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2"]
Jul 17 22:38:15.263079 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:15.263034    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-bxsq2"]
Jul 17 22:38:17.182885 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:17.182850    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47b2778f-99a2-47a6-822b-4020f4e78c5b" path="/var/lib/kubelet/pods/47b2778f-99a2-47a6-822b-4020f4e78c5b/volumes"
Jul 17 22:38:18.655700 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:18.655626    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:38:25.306754 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.306711    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm"]
Jul 17 22:38:25.307238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307084    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d68724b-39ba-404c-83ee-1703da3583ec" containerName="revoke-keys"
Jul 17 22:38:25.307238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307096    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d68724b-39ba-404c-83ee-1703da3583ec" containerName="revoke-keys"
Jul 17 22:38:25.307238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307110    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0edce941-b48a-4428-96be-73c500f726d9" containerName="revoke-keys"
Jul 17 22:38:25.307238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307116    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0edce941-b48a-4428-96be-73c500f726d9" containerName="revoke-keys"
Jul 17 22:38:25.307238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307125    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b2778f-99a2-47a6-822b-4020f4e78c5b" containerName="revoke-keys"
Jul 17 22:38:25.307238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307130    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b2778f-99a2-47a6-822b-4020f4e78c5b" containerName="revoke-keys"
Jul 17 22:38:25.307238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307140    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b2778f-99a2-47a6-822b-4020f4e78c5b" containerName="revoke-keys"
Jul 17 22:38:25.307238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307146    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b2778f-99a2-47a6-822b-4020f4e78c5b" containerName="revoke-keys"
Jul 17 22:38:25.307238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307154    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b2778f-99a2-47a6-822b-4020f4e78c5b" containerName="revoke-keys"
Jul 17 22:38:25.307238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307160    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b2778f-99a2-47a6-822b-4020f4e78c5b" containerName="revoke-keys"
Jul 17 22:38:25.307238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307167    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d68724b-39ba-404c-83ee-1703da3583ec" containerName="revoke-keys"
Jul 17 22:38:25.307238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307172    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d68724b-39ba-404c-83ee-1703da3583ec" containerName="revoke-keys"
Jul 17 22:38:25.307238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307224    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d68724b-39ba-404c-83ee-1703da3583ec" containerName="revoke-keys"
Jul 17 22:38:25.307238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307232    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b2778f-99a2-47a6-822b-4020f4e78c5b" containerName="revoke-keys"
Jul 17 22:38:25.307238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307238    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d68724b-39ba-404c-83ee-1703da3583ec" containerName="revoke-keys"
Jul 17 22:38:25.307238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307243    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0edce941-b48a-4428-96be-73c500f726d9" containerName="revoke-keys"
Jul 17 22:38:25.307754 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.307249    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b2778f-99a2-47a6-822b-4020f4e78c5b" containerName="revoke-keys"
Jul 17 22:38:25.310489 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.310472    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm"
Jul 17 22:38:25.313162 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.313134    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:38:25.313299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.313135    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:38:25.316877 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.316848    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm"]
Jul 17 22:38:25.364339 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.364303    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-whnrm\" (UID: \"ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm"
Jul 17 22:38:25.465488 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.465451    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-whnrm\" (UID: \"ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm"
Jul 17 22:38:25.466131 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.466110    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-whnrm\" (UID: \"ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm"
Jul 17 22:38:25.623560 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.623471    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm"
Jul 17 22:38:25.764859 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:25.764829    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm"]
Jul 17 22:38:25.767440 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:38:25.767410    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccaecb5b_0a4e_423a_80a7_ae6ffa5084fa.slice/crio-b5fd663bd2f1853c901da1e6641ccf4a1aef0308c1e1cffe1f6fa6d94acb93cc WatchSource:0}: Error finding container b5fd663bd2f1853c901da1e6641ccf4a1aef0308c1e1cffe1f6fa6d94acb93cc: Status 404 returned error can't find the container with id b5fd663bd2f1853c901da1e6641ccf4a1aef0308c1e1cffe1f6fa6d94acb93cc
Jul 17 22:38:26.302789 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:26.302753    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm" event={"ID":"ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa","Type":"ContainerStarted","Data":"d59d6e9d96af460e24554e0a01c6f021ae79968f906bc645d19ead771b78787c"}
Jul 17 22:38:26.302980 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:26.302803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm" event={"ID":"ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa","Type":"ContainerStarted","Data":"b5fd663bd2f1853c901da1e6641ccf4a1aef0308c1e1cffe1f6fa6d94acb93cc"}
Jul 17 22:38:26.319313 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:26.319251    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm" podStartSLOduration=1.319235444 podStartE2EDuration="1.319235444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:38:26.317476261 +0000 UTC m=+3169.706492913" watchObservedRunningTime="2026-07-17 22:38:26.319235444 +0000 UTC m=+3169.708252096"
Jul 17 22:38:26.811869 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:26.811834    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7"]
Jul 17 22:38:26.812236 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:26.812223    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d68724b-39ba-404c-83ee-1703da3583ec" containerName="revoke-keys"
Jul 17 22:38:26.812287 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:26.812238    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d68724b-39ba-404c-83ee-1703da3583ec" containerName="revoke-keys"
Jul 17 22:38:26.812323 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:26.812306    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d68724b-39ba-404c-83ee-1703da3583ec" containerName="revoke-keys"
Jul 17 22:38:26.812323 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:26.812315    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b2778f-99a2-47a6-822b-4020f4e78c5b" containerName="revoke-keys"
Jul 17 22:38:26.815342 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:26.815325    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7"
Jul 17 22:38:26.824455 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:26.824430    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7"]
Jul 17 22:38:26.878568 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:26.878531    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2807bc4-48b6-4fda-8148-ff9568d8c9eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-g84l7\" (UID: \"c2807bc4-48b6-4fda-8148-ff9568d8c9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7"
Jul 17 22:38:26.979752 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:26.979709    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2807bc4-48b6-4fda-8148-ff9568d8c9eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-g84l7\" (UID: \"c2807bc4-48b6-4fda-8148-ff9568d8c9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7"
Jul 17 22:38:26.980439 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:26.980410    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2807bc4-48b6-4fda-8148-ff9568d8c9eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-g84l7\" (UID: \"c2807bc4-48b6-4fda-8148-ff9568d8c9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7"
Jul 17 22:38:27.127896 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:27.127794    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7"
Jul 17 22:38:27.272610 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:27.272583    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7"]
Jul 17 22:38:27.274480 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:38:27.274442    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2807bc4_48b6_4fda_8148_ff9568d8c9eb.slice/crio-152e9ac42c11e25d814887e7d909dc39b98719651448eaedec290fdd92686c24 WatchSource:0}: Error finding container 152e9ac42c11e25d814887e7d909dc39b98719651448eaedec290fdd92686c24: Status 404 returned error can't find the container with id 152e9ac42c11e25d814887e7d909dc39b98719651448eaedec290fdd92686c24
Jul 17 22:38:27.308873 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:27.308839    2565 generic.go:356] "Generic (PLEG): container finished" podID="ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa" containerID="d59d6e9d96af460e24554e0a01c6f021ae79968f906bc645d19ead771b78787c" exitCode=6
Jul 17 22:38:27.309048 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:27.308930    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm" event={"ID":"ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa","Type":"ContainerDied","Data":"d59d6e9d96af460e24554e0a01c6f021ae79968f906bc645d19ead771b78787c"}
Jul 17 22:38:27.309144 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:27.309104    2565 scope.go:117] "RemoveContainer" containerID="d59d6e9d96af460e24554e0a01c6f021ae79968f906bc645d19ead771b78787c"
Jul 17 22:38:27.310247 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:27.310215    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7" event={"ID":"c2807bc4-48b6-4fda-8148-ff9568d8c9eb","Type":"ContainerStarted","Data":"152e9ac42c11e25d814887e7d909dc39b98719651448eaedec290fdd92686c24"}
Jul 17 22:38:28.317430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:28.317395    2565 generic.go:356] "Generic (PLEG): container finished" podID="ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa" containerID="7d9277cc61ae8b02ce3b9d5f9a04f38789bf896cb0574a7127684c1d4598f43d" exitCode=6
Jul 17 22:38:28.317884 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:28.317477    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm" event={"ID":"ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa","Type":"ContainerDied","Data":"7d9277cc61ae8b02ce3b9d5f9a04f38789bf896cb0574a7127684c1d4598f43d"}
Jul 17 22:38:28.317884 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:28.317535    2565 scope.go:117] "RemoveContainer" containerID="d59d6e9d96af460e24554e0a01c6f021ae79968f906bc645d19ead771b78787c"
Jul 17 22:38:28.317884 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:28.317722    2565 scope.go:117] "RemoveContainer" containerID="7d9277cc61ae8b02ce3b9d5f9a04f38789bf896cb0574a7127684c1d4598f43d"
Jul 17 22:38:28.318024 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:38:28.317962    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-whnrm_odh-ai-gateway-infra(ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm" podUID="ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa"
Jul 17 22:38:28.319226 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:28.319203    2565 generic.go:356] "Generic (PLEG): container finished" podID="c2807bc4-48b6-4fda-8148-ff9568d8c9eb" containerID="1b4e33270ed59fae381841769cc98b3c1ac06aed622188384205c9183296f418" exitCode=6
Jul 17 22:38:28.319332 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:28.319232    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7" event={"ID":"c2807bc4-48b6-4fda-8148-ff9568d8c9eb","Type":"ContainerDied","Data":"1b4e33270ed59fae381841769cc98b3c1ac06aed622188384205c9183296f418"}
Jul 17 22:38:28.319464 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:28.319451    2565 scope.go:117] "RemoveContainer" containerID="1b4e33270ed59fae381841769cc98b3c1ac06aed622188384205c9183296f418"
Jul 17 22:38:28.645612 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:28.645579    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:38:29.325806 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:29.325779    2565 scope.go:117] "RemoveContainer" containerID="7d9277cc61ae8b02ce3b9d5f9a04f38789bf896cb0574a7127684c1d4598f43d"
Jul 17 22:38:29.326297 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:38:29.326053    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-whnrm_odh-ai-gateway-infra(ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm" podUID="ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa"
Jul 17 22:38:29.327524 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:29.327501    2565 generic.go:356] "Generic (PLEG): container finished" podID="c2807bc4-48b6-4fda-8148-ff9568d8c9eb" containerID="eba65ab248744bcc814d4baef0ae3c85af3c5c47f600d6754373c2abfb4ae31a" exitCode=6
Jul 17 22:38:29.327648 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:29.327544    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7" event={"ID":"c2807bc4-48b6-4fda-8148-ff9568d8c9eb","Type":"ContainerDied","Data":"eba65ab248744bcc814d4baef0ae3c85af3c5c47f600d6754373c2abfb4ae31a"}
Jul 17 22:38:29.327648 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:29.327584    2565 scope.go:117] "RemoveContainer" containerID="1b4e33270ed59fae381841769cc98b3c1ac06aed622188384205c9183296f418"
Jul 17 22:38:29.327790 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:29.327771    2565 scope.go:117] "RemoveContainer" containerID="eba65ab248744bcc814d4baef0ae3c85af3c5c47f600d6754373c2abfb4ae31a"
Jul 17 22:38:29.328015 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:38:29.327988    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-g84l7_odh-ai-gateway-infra(c2807bc4-48b6-4fda-8148-ff9568d8c9eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7" podUID="c2807bc4-48b6-4fda-8148-ff9568d8c9eb"
Jul 17 22:38:30.334199 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:30.334163    2565 scope.go:117] "RemoveContainer" containerID="eba65ab248744bcc814d4baef0ae3c85af3c5c47f600d6754373c2abfb4ae31a"
Jul 17 22:38:30.334684 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:38:30.334477    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-g84l7_odh-ai-gateway-infra(c2807bc4-48b6-4fda-8148-ff9568d8c9eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7" podUID="c2807bc4-48b6-4fda-8148-ff9568d8c9eb"
Jul 17 22:38:38.621663 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:38.621617    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt"]
Jul 17 22:38:38.626974 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:38.626950    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt"
Jul 17 22:38:38.638880 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:38.638832    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt"]
Jul 17 22:38:38.695838 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:38.695795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f09421-593e-4a5e-9744-3cbb8a2ea7e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt\" (UID: \"49f09421-593e-4a5e-9744-3cbb8a2ea7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt"
Jul 17 22:38:38.796569 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:38.796523    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f09421-593e-4a5e-9744-3cbb8a2ea7e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt\" (UID: \"49f09421-593e-4a5e-9744-3cbb8a2ea7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt"
Jul 17 22:38:38.797225 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:38.797203    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f09421-593e-4a5e-9744-3cbb8a2ea7e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt\" (UID: \"49f09421-593e-4a5e-9744-3cbb8a2ea7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt"
Jul 17 22:38:38.941886 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:38.941799    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt"
Jul 17 22:38:39.082681 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:39.082649    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt"]
Jul 17 22:38:39.084691 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:38:39.084662    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f09421_593e_4a5e_9744_3cbb8a2ea7e7.slice/crio-57fa02f88b5a8e632151d65f11eb16f065c5606c4a402eb510367738a61b74bd WatchSource:0}: Error finding container 57fa02f88b5a8e632151d65f11eb16f065c5606c4a402eb510367738a61b74bd: Status 404 returned error can't find the container with id 57fa02f88b5a8e632151d65f11eb16f065c5606c4a402eb510367738a61b74bd
Jul 17 22:38:39.379995 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:39.379949    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt" event={"ID":"49f09421-593e-4a5e-9744-3cbb8a2ea7e7","Type":"ContainerStarted","Data":"e108b2e3fbf0d6fc3744b9f1c3b2a9c98a163cc1704fb8b63e66734068a35f06"}
Jul 17 22:38:39.380189 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:39.380009    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt" event={"ID":"49f09421-593e-4a5e-9744-3cbb8a2ea7e7","Type":"ContainerStarted","Data":"57fa02f88b5a8e632151d65f11eb16f065c5606c4a402eb510367738a61b74bd"}
Jul 17 22:38:39.395958 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:39.395902    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt" podStartSLOduration=1.395885478 podStartE2EDuration="1.395885478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:38:39.395358877 +0000 UTC m=+3182.784375525" watchObservedRunningTime="2026-07-17 22:38:39.395885478 +0000 UTC m=+3182.784902126"
Jul 17 22:38:40.387256 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:40.387223    2565 generic.go:356] "Generic (PLEG): container finished" podID="49f09421-593e-4a5e-9744-3cbb8a2ea7e7" containerID="e108b2e3fbf0d6fc3744b9f1c3b2a9c98a163cc1704fb8b63e66734068a35f06" exitCode=6
Jul 17 22:38:40.387743 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:40.387304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt" event={"ID":"49f09421-593e-4a5e-9744-3cbb8a2ea7e7","Type":"ContainerDied","Data":"e108b2e3fbf0d6fc3744b9f1c3b2a9c98a163cc1704fb8b63e66734068a35f06"}
Jul 17 22:38:40.387743 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:40.387574    2565 scope.go:117] "RemoveContainer" containerID="e108b2e3fbf0d6fc3744b9f1c3b2a9c98a163cc1704fb8b63e66734068a35f06"
Jul 17 22:38:41.176600 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:41.176571    2565 scope.go:117] "RemoveContainer" containerID="eba65ab248744bcc814d4baef0ae3c85af3c5c47f600d6754373c2abfb4ae31a"
Jul 17 22:38:41.395694 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:41.395666    2565 generic.go:356] "Generic (PLEG): container finished" podID="49f09421-593e-4a5e-9744-3cbb8a2ea7e7" containerID="97b426bcd43fe042f3406d8f5c931b5d3a7411d3d6fda4e7899aa5f36f582822" exitCode=6
Jul 17 22:38:41.396061 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:41.395752    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt" event={"ID":"49f09421-593e-4a5e-9744-3cbb8a2ea7e7","Type":"ContainerDied","Data":"97b426bcd43fe042f3406d8f5c931b5d3a7411d3d6fda4e7899aa5f36f582822"}
Jul 17 22:38:41.396061 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:41.395818    2565 scope.go:117] "RemoveContainer" containerID="e108b2e3fbf0d6fc3744b9f1c3b2a9c98a163cc1704fb8b63e66734068a35f06"
Jul 17 22:38:41.396205 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:41.396091    2565 scope.go:117] "RemoveContainer" containerID="97b426bcd43fe042f3406d8f5c931b5d3a7411d3d6fda4e7899aa5f36f582822"
Jul 17 22:38:41.396356 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:38:41.396334    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt_odh-ai-gateway-infra(49f09421-593e-4a5e-9744-3cbb8a2ea7e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt" podUID="49f09421-593e-4a5e-9744-3cbb8a2ea7e7"
Jul 17 22:38:42.402743 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:42.402706    2565 generic.go:356] "Generic (PLEG): container finished" podID="c2807bc4-48b6-4fda-8148-ff9568d8c9eb" containerID="dd611e030a6ba804ec0fb12f9fe92c9cc80565619d8590ff5593f3d46236e5f2" exitCode=6
Jul 17 22:38:42.403265 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:42.402784    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7" event={"ID":"c2807bc4-48b6-4fda-8148-ff9568d8c9eb","Type":"ContainerDied","Data":"dd611e030a6ba804ec0fb12f9fe92c9cc80565619d8590ff5593f3d46236e5f2"}
Jul 17 22:38:42.403265 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:42.402840    2565 scope.go:117] "RemoveContainer" containerID="eba65ab248744bcc814d4baef0ae3c85af3c5c47f600d6754373c2abfb4ae31a"
Jul 17 22:38:42.403265 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:42.403199    2565 scope.go:117] "RemoveContainer" containerID="dd611e030a6ba804ec0fb12f9fe92c9cc80565619d8590ff5593f3d46236e5f2"
Jul 17 22:38:42.403488 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:38:42.403465    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-g84l7_odh-ai-gateway-infra(c2807bc4-48b6-4fda-8148-ff9568d8c9eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7" podUID="c2807bc4-48b6-4fda-8148-ff9568d8c9eb"
Jul 17 22:38:42.404807 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:42.404788    2565 scope.go:117] "RemoveContainer" containerID="97b426bcd43fe042f3406d8f5c931b5d3a7411d3d6fda4e7899aa5f36f582822"
Jul 17 22:38:42.405000 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:38:42.404983    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt_odh-ai-gateway-infra(49f09421-593e-4a5e-9744-3cbb8a2ea7e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt" podUID="49f09421-593e-4a5e-9744-3cbb8a2ea7e7"
Jul 17 22:38:43.430865 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:43.430830    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7"]
Jul 17 22:38:43.576305 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:43.576282    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7"
Jul 17 22:38:43.731113 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:43.731024    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:38:43.743105 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:43.743061    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2807bc4-48b6-4fda-8148-ff9568d8c9eb-maas-api-service-ca\") pod \"c2807bc4-48b6-4fda-8148-ff9568d8c9eb\" (UID: \"c2807bc4-48b6-4fda-8148-ff9568d8c9eb\") "
Jul 17 22:38:43.743361 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:43.743340    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2807bc4-48b6-4fda-8148-ff9568d8c9eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2807bc4-48b6-4fda-8148-ff9568d8c9eb" (UID: "c2807bc4-48b6-4fda-8148-ff9568d8c9eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:38:43.844515 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:43.844481    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2807bc4-48b6-4fda-8148-ff9568d8c9eb-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:38:44.177176 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:44.177127    2565 scope.go:117] "RemoveContainer" containerID="7d9277cc61ae8b02ce3b9d5f9a04f38789bf896cb0574a7127684c1d4598f43d"
Jul 17 22:38:44.419195 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:44.419167    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7"
Jul 17 22:38:44.419389 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:44.419209    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7" event={"ID":"c2807bc4-48b6-4fda-8148-ff9568d8c9eb","Type":"ContainerDied","Data":"152e9ac42c11e25d814887e7d909dc39b98719651448eaedec290fdd92686c24"}
Jul 17 22:38:44.419389 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:44.419271    2565 scope.go:117] "RemoveContainer" containerID="dd611e030a6ba804ec0fb12f9fe92c9cc80565619d8590ff5593f3d46236e5f2"
Jul 17 22:38:44.453020 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:44.452989    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7"]
Jul 17 22:38:44.469433 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:44.469409    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-g84l7"]
Jul 17 22:38:44.679848 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:44.679765    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:38:45.182317 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:45.182281    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2807bc4-48b6-4fda-8148-ff9568d8c9eb" path="/var/lib/kubelet/pods/c2807bc4-48b6-4fda-8148-ff9568d8c9eb/volumes"
Jul 17 22:38:45.426774 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:45.426743    2565 generic.go:356] "Generic (PLEG): container finished" podID="ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa" containerID="005d17f474926f6021333c2da9ace5b9c2f6e30e951aacebd709c90af66bd684" exitCode=6
Jul 17 22:38:45.426922 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:45.426781    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm" event={"ID":"ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa","Type":"ContainerDied","Data":"005d17f474926f6021333c2da9ace5b9c2f6e30e951aacebd709c90af66bd684"}
Jul 17 22:38:45.426922 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:45.426861    2565 scope.go:117] "RemoveContainer" containerID="7d9277cc61ae8b02ce3b9d5f9a04f38789bf896cb0574a7127684c1d4598f43d"
Jul 17 22:38:45.427145 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:45.427124    2565 scope.go:117] "RemoveContainer" containerID="005d17f474926f6021333c2da9ace5b9c2f6e30e951aacebd709c90af66bd684"
Jul 17 22:38:45.427403 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:38:45.427381    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-whnrm_odh-ai-gateway-infra(ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm" podUID="ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa"
Jul 17 22:38:46.460618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:46.460580    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm"]
Jul 17 22:38:46.593542 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:46.593516    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm"
Jul 17 22:38:46.771478 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:46.771445    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa-maas-api-service-ca\") pod \"ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa\" (UID: \"ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa\") "
Jul 17 22:38:46.771825 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:46.771802    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa" (UID: "ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:38:46.873119 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:46.873062    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:38:47.364545 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:47.364506    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:38:47.439837 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:47.439760    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm"
Jul 17 22:38:47.439975 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:47.439758    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm" event={"ID":"ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa","Type":"ContainerDied","Data":"b5fd663bd2f1853c901da1e6641ccf4a1aef0308c1e1cffe1f6fa6d94acb93cc"}
Jul 17 22:38:47.439975 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:47.439896    2565 scope.go:117] "RemoveContainer" containerID="005d17f474926f6021333c2da9ace5b9c2f6e30e951aacebd709c90af66bd684"
Jul 17 22:38:47.461051 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:47.461018    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm"]
Jul 17 22:38:47.469826 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:47.469793    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-whnrm"]
Jul 17 22:38:49.182817 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:49.182773    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa" path="/var/lib/kubelet/pods/ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa/volumes"
Jul 17 22:38:51.868078 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:51.868032    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:38:53.211732 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.211696    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c"]
Jul 17 22:38:53.212159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.212050    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2807bc4-48b6-4fda-8148-ff9568d8c9eb" containerName="revoke-keys"
Jul 17 22:38:53.212159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.212061    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2807bc4-48b6-4fda-8148-ff9568d8c9eb" containerName="revoke-keys"
Jul 17 22:38:53.212159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.212090    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa" containerName="revoke-keys"
Jul 17 22:38:53.212159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.212096    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa" containerName="revoke-keys"
Jul 17 22:38:53.212159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.212110    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa" containerName="revoke-keys"
Jul 17 22:38:53.212159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.212116    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa" containerName="revoke-keys"
Jul 17 22:38:53.212159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.212123    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa" containerName="revoke-keys"
Jul 17 22:38:53.212159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.212128    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa" containerName="revoke-keys"
Jul 17 22:38:53.212159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.212134    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2807bc4-48b6-4fda-8148-ff9568d8c9eb" containerName="revoke-keys"
Jul 17 22:38:53.212159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.212139    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2807bc4-48b6-4fda-8148-ff9568d8c9eb" containerName="revoke-keys"
Jul 17 22:38:53.212461 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.212212    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2807bc4-48b6-4fda-8148-ff9568d8c9eb" containerName="revoke-keys"
Jul 17 22:38:53.212461 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.212219    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa" containerName="revoke-keys"
Jul 17 22:38:53.212461 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.212225    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2807bc4-48b6-4fda-8148-ff9568d8c9eb" containerName="revoke-keys"
Jul 17 22:38:53.212461 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.212233    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa" containerName="revoke-keys"
Jul 17 22:38:53.212461 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.212238    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2807bc4-48b6-4fda-8148-ff9568d8c9eb" containerName="revoke-keys"
Jul 17 22:38:53.212461 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.212244    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccaecb5b-0a4e-423a-80a7-ae6ffa5084fa" containerName="revoke-keys"
Jul 17 22:38:53.216567 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.216550    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c"
Jul 17 22:38:53.222793 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.222758    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c"]
Jul 17 22:38:53.336249 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.336211    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45743089-59fb-449e-b289-771bb6ed5705-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-fng9c\" (UID: \"45743089-59fb-449e-b289-771bb6ed5705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c"
Jul 17 22:38:53.437138 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.437095    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45743089-59fb-449e-b289-771bb6ed5705-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-fng9c\" (UID: \"45743089-59fb-449e-b289-771bb6ed5705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c"
Jul 17 22:38:53.437741 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.437715    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45743089-59fb-449e-b289-771bb6ed5705-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-fng9c\" (UID: \"45743089-59fb-449e-b289-771bb6ed5705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c"
Jul 17 22:38:53.529143 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.529109    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c"
Jul 17 22:38:53.691904 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:53.691877    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c"]
Jul 17 22:38:53.695269 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:38:53.695236    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45743089_59fb_449e_b289_771bb6ed5705.slice/crio-5efc90f76d8432b62cee3c088873a0d5c3f38f49dbb4777320c7f953c90eb4ee WatchSource:0}: Error finding container 5efc90f76d8432b62cee3c088873a0d5c3f38f49dbb4777320c7f953c90eb4ee: Status 404 returned error can't find the container with id 5efc90f76d8432b62cee3c088873a0d5c3f38f49dbb4777320c7f953c90eb4ee
Jul 17 22:38:54.479309 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:54.479270    2565 generic.go:356] "Generic (PLEG): container finished" podID="45743089-59fb-449e-b289-771bb6ed5705" containerID="57cea0240975d900c1bfe10dbb43c57866d3bafd91e2c99f5315b91864b21530" exitCode=6
Jul 17 22:38:54.479679 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:54.479328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c" event={"ID":"45743089-59fb-449e-b289-771bb6ed5705","Type":"ContainerDied","Data":"57cea0240975d900c1bfe10dbb43c57866d3bafd91e2c99f5315b91864b21530"}
Jul 17 22:38:54.479679 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:54.479412    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c" event={"ID":"45743089-59fb-449e-b289-771bb6ed5705","Type":"ContainerStarted","Data":"5efc90f76d8432b62cee3c088873a0d5c3f38f49dbb4777320c7f953c90eb4ee"}
Jul 17 22:38:54.479679 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:54.479663    2565 scope.go:117] "RemoveContainer" containerID="57cea0240975d900c1bfe10dbb43c57866d3bafd91e2c99f5315b91864b21530"
Jul 17 22:38:55.489707 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:55.488947    2565 generic.go:356] "Generic (PLEG): container finished" podID="45743089-59fb-449e-b289-771bb6ed5705" containerID="b5028d6b4c60e6d8de978112e7ab947106b521f9d9b49b4a3dc57c12518cb47b" exitCode=6
Jul 17 22:38:55.489707 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:55.489031    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c" event={"ID":"45743089-59fb-449e-b289-771bb6ed5705","Type":"ContainerDied","Data":"b5028d6b4c60e6d8de978112e7ab947106b521f9d9b49b4a3dc57c12518cb47b"}
Jul 17 22:38:55.489707 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:55.489535    2565 scope.go:117] "RemoveContainer" containerID="57cea0240975d900c1bfe10dbb43c57866d3bafd91e2c99f5315b91864b21530"
Jul 17 22:38:55.490214 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:55.489867    2565 scope.go:117] "RemoveContainer" containerID="b5028d6b4c60e6d8de978112e7ab947106b521f9d9b49b4a3dc57c12518cb47b"
Jul 17 22:38:55.490214 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:38:55.490160    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-fng9c_odh-ai-gateway-infra(45743089-59fb-449e-b289-771bb6ed5705)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c" podUID="45743089-59fb-449e-b289-771bb6ed5705"
Jul 17 22:38:56.176481 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:56.176445    2565 scope.go:117] "RemoveContainer" containerID="97b426bcd43fe042f3406d8f5c931b5d3a7411d3d6fda4e7899aa5f36f582822"
Jul 17 22:38:56.495928 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:56.495892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt" event={"ID":"49f09421-593e-4a5e-9744-3cbb8a2ea7e7","Type":"ContainerStarted","Data":"851967f37282e21a34623b42c021c7d824d5a8658033a200fa3b9ea7e0fdc49f"}
Jul 17 22:38:56.497636 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:56.497616    2565 scope.go:117] "RemoveContainer" containerID="b5028d6b4c60e6d8de978112e7ab947106b521f9d9b49b4a3dc57c12518cb47b"
Jul 17 22:38:56.497844 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:38:56.497826    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-fng9c_odh-ai-gateway-infra(45743089-59fb-449e-b289-771bb6ed5705)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c" podUID="45743089-59fb-449e-b289-771bb6ed5705"
Jul 17 22:38:57.504825 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:57.504788    2565 generic.go:356] "Generic (PLEG): container finished" podID="49f09421-593e-4a5e-9744-3cbb8a2ea7e7" containerID="851967f37282e21a34623b42c021c7d824d5a8658033a200fa3b9ea7e0fdc49f" exitCode=6
Jul 17 22:38:57.505345 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:57.504860    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt" event={"ID":"49f09421-593e-4a5e-9744-3cbb8a2ea7e7","Type":"ContainerDied","Data":"851967f37282e21a34623b42c021c7d824d5a8658033a200fa3b9ea7e0fdc49f"}
Jul 17 22:38:57.505345 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:57.504921    2565 scope.go:117] "RemoveContainer" containerID="97b426bcd43fe042f3406d8f5c931b5d3a7411d3d6fda4e7899aa5f36f582822"
Jul 17 22:38:57.505345 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:57.505240    2565 scope.go:117] "RemoveContainer" containerID="851967f37282e21a34623b42c021c7d824d5a8658033a200fa3b9ea7e0fdc49f"
Jul 17 22:38:57.505549 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:38:57.505528    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt_odh-ai-gateway-infra(49f09421-593e-4a5e-9744-3cbb8a2ea7e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt" podUID="49f09421-593e-4a5e-9744-3cbb8a2ea7e7"
Jul 17 22:38:57.556119 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:57.556060    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt"]
Jul 17 22:38:58.650591 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:58.650567    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt"
Jul 17 22:38:58.680646 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:58.680616    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f09421-593e-4a5e-9744-3cbb8a2ea7e7-maas-api-service-ca\") pod \"49f09421-593e-4a5e-9744-3cbb8a2ea7e7\" (UID: \"49f09421-593e-4a5e-9744-3cbb8a2ea7e7\") "
Jul 17 22:38:58.680952 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:58.680930    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f09421-593e-4a5e-9744-3cbb8a2ea7e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "49f09421-593e-4a5e-9744-3cbb8a2ea7e7" (UID: "49f09421-593e-4a5e-9744-3cbb8a2ea7e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:38:58.782107 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:58.781987    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49f09421-593e-4a5e-9744-3cbb8a2ea7e7-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:38:59.518857 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:59.518832    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt"
Jul 17 22:38:59.519045 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:59.518831    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt" event={"ID":"49f09421-593e-4a5e-9744-3cbb8a2ea7e7","Type":"ContainerDied","Data":"57fa02f88b5a8e632151d65f11eb16f065c5606c4a402eb510367738a61b74bd"}
Jul 17 22:38:59.519045 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:59.518965    2565 scope.go:117] "RemoveContainer" containerID="851967f37282e21a34623b42c021c7d824d5a8658033a200fa3b9ea7e0fdc49f"
Jul 17 22:38:59.538226 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:59.538201    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt"]
Jul 17 22:38:59.542467 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:38:59.542444    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-2pxjt"]
Jul 17 22:39:01.182870 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:01.182835    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49f09421-593e-4a5e-9744-3cbb8a2ea7e7" path="/var/lib/kubelet/pods/49f09421-593e-4a5e-9744-3cbb8a2ea7e7/volumes"
Jul 17 22:39:02.943474 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:02.943434    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:39:06.315135 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:06.315102    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:39:06.420908 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:06.420875    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:39:08.177051 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:08.177017    2565 scope.go:117] "RemoveContainer" containerID="b5028d6b4c60e6d8de978112e7ab947106b521f9d9b49b4a3dc57c12518cb47b"
Jul 17 22:39:08.567269 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:08.567239    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c" event={"ID":"45743089-59fb-449e-b289-771bb6ed5705","Type":"ContainerStarted","Data":"e79d90593fae414fde4785ac2a31648adfaae0862b4077bb02b5930bc009c763"}
Jul 17 22:39:08.585364 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:08.585314    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c" podStartSLOduration=15.585299066 podStartE2EDuration="15.585299066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:39:08.584304856 +0000 UTC m=+3211.973321507" watchObservedRunningTime="2026-07-17 22:39:08.585299066 +0000 UTC m=+3211.974315718"
Jul 17 22:39:09.574333 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:09.574303    2565 generic.go:356] "Generic (PLEG): container finished" podID="45743089-59fb-449e-b289-771bb6ed5705" containerID="e79d90593fae414fde4785ac2a31648adfaae0862b4077bb02b5930bc009c763" exitCode=6
Jul 17 22:39:09.574729 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:09.574373    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c" event={"ID":"45743089-59fb-449e-b289-771bb6ed5705","Type":"ContainerDied","Data":"e79d90593fae414fde4785ac2a31648adfaae0862b4077bb02b5930bc009c763"}
Jul 17 22:39:09.574729 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:09.574439    2565 scope.go:117] "RemoveContainer" containerID="b5028d6b4c60e6d8de978112e7ab947106b521f9d9b49b4a3dc57c12518cb47b"
Jul 17 22:39:09.574812 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:09.574743    2565 scope.go:117] "RemoveContainer" containerID="e79d90593fae414fde4785ac2a31648adfaae0862b4077bb02b5930bc009c763"
Jul 17 22:39:09.575029 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:39:09.575006    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-fng9c_odh-ai-gateway-infra(45743089-59fb-449e-b289-771bb6ed5705)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c" podUID="45743089-59fb-449e-b289-771bb6ed5705"
Jul 17 22:39:09.602928 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:09.602889    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c"]
Jul 17 22:39:10.718980 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:10.718954    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c"
Jul 17 22:39:10.798609 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:10.798570    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45743089-59fb-449e-b289-771bb6ed5705-maas-api-service-ca\") pod \"45743089-59fb-449e-b289-771bb6ed5705\" (UID: \"45743089-59fb-449e-b289-771bb6ed5705\") "
Jul 17 22:39:10.798939 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:10.798918    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45743089-59fb-449e-b289-771bb6ed5705-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "45743089-59fb-449e-b289-771bb6ed5705" (UID: "45743089-59fb-449e-b289-771bb6ed5705"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:39:10.899453 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:10.899360    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45743089-59fb-449e-b289-771bb6ed5705-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:39:11.588598 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:11.588572    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c"
Jul 17 22:39:11.588792 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:11.588617    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c" event={"ID":"45743089-59fb-449e-b289-771bb6ed5705","Type":"ContainerDied","Data":"5efc90f76d8432b62cee3c088873a0d5c3f38f49dbb4777320c7f953c90eb4ee"}
Jul 17 22:39:11.588792 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:11.588679    2565 scope.go:117] "RemoveContainer" containerID="e79d90593fae414fde4785ac2a31648adfaae0862b4077bb02b5930bc009c763"
Jul 17 22:39:11.605722 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:11.605692    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c"]
Jul 17 22:39:11.614861 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:11.614832    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-fng9c"]
Jul 17 22:39:13.182279 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:13.182245    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45743089-59fb-449e-b289-771bb6ed5705" path="/var/lib/kubelet/pods/45743089-59fb-449e-b289-771bb6ed5705/volumes"
Jul 17 22:39:16.857554 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.857514    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2"]
Jul 17 22:39:16.858052 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.857870    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49f09421-593e-4a5e-9744-3cbb8a2ea7e7" containerName="revoke-keys"
Jul 17 22:39:16.858052 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.857880    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="49f09421-593e-4a5e-9744-3cbb8a2ea7e7" containerName="revoke-keys"
Jul 17 22:39:16.858052 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.857899    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2807bc4-48b6-4fda-8148-ff9568d8c9eb" containerName="revoke-keys"
Jul 17 22:39:16.858052 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.857906    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2807bc4-48b6-4fda-8148-ff9568d8c9eb" containerName="revoke-keys"
Jul 17 22:39:16.858052 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.857911    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45743089-59fb-449e-b289-771bb6ed5705" containerName="revoke-keys"
Jul 17 22:39:16.858052 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.857918    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="45743089-59fb-449e-b289-771bb6ed5705" containerName="revoke-keys"
Jul 17 22:39:16.858052 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.857930    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49f09421-593e-4a5e-9744-3cbb8a2ea7e7" containerName="revoke-keys"
Jul 17 22:39:16.858052 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.857935    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="49f09421-593e-4a5e-9744-3cbb8a2ea7e7" containerName="revoke-keys"
Jul 17 22:39:16.858052 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.857941    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45743089-59fb-449e-b289-771bb6ed5705" containerName="revoke-keys"
Jul 17 22:39:16.858052 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.857945    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="45743089-59fb-449e-b289-771bb6ed5705" containerName="revoke-keys"
Jul 17 22:39:16.858052 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.857998    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="45743089-59fb-449e-b289-771bb6ed5705" containerName="revoke-keys"
Jul 17 22:39:16.858052 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.858004    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="49f09421-593e-4a5e-9744-3cbb8a2ea7e7" containerName="revoke-keys"
Jul 17 22:39:16.858052 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.858013    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="49f09421-593e-4a5e-9744-3cbb8a2ea7e7" containerName="revoke-keys"
Jul 17 22:39:16.858052 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.858020    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="45743089-59fb-449e-b289-771bb6ed5705" containerName="revoke-keys"
Jul 17 22:39:16.862496 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.862474    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2"
Jul 17 22:39:16.865410 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.865392    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:39:16.865523 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.865438    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:39:16.871117 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.871089    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2"]
Jul 17 22:39:16.952858 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:16.952813    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d5d6da-bc99-487e-959d-0f05e4525d27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2\" (UID: \"88d5d6da-bc99-487e-959d-0f05e4525d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2"
Jul 17 22:39:17.053779 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:17.053727    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d5d6da-bc99-487e-959d-0f05e4525d27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2\" (UID: \"88d5d6da-bc99-487e-959d-0f05e4525d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2"
Jul 17 22:39:17.054395 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:17.054376    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d5d6da-bc99-487e-959d-0f05e4525d27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2\" (UID: \"88d5d6da-bc99-487e-959d-0f05e4525d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2"
Jul 17 22:39:17.174667 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:17.174569    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2"
Jul 17 22:39:17.318724 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:17.318697    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2"]
Jul 17 22:39:17.320532 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:39:17.320504    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d5d6da_bc99_487e_959d_0f05e4525d27.slice/crio-c93b46c74d01d3fc8721454c0b1ff39e01e8f8b743bb863eec2923c31614cd03 WatchSource:0}: Error finding container c93b46c74d01d3fc8721454c0b1ff39e01e8f8b743bb863eec2923c31614cd03: Status 404 returned error can't find the container with id c93b46c74d01d3fc8721454c0b1ff39e01e8f8b743bb863eec2923c31614cd03
Jul 17 22:39:17.629505 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:17.629463    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2" event={"ID":"88d5d6da-bc99-487e-959d-0f05e4525d27","Type":"ContainerStarted","Data":"c93b46c74d01d3fc8721454c0b1ff39e01e8f8b743bb863eec2923c31614cd03"}
Jul 17 22:39:18.636179 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:18.636143    2565 generic.go:356] "Generic (PLEG): container finished" podID="88d5d6da-bc99-487e-959d-0f05e4525d27" containerID="cc9452927557f9cc81574af03549bc0b43904f480d79a4cc91050994e0d04152" exitCode=6
Jul 17 22:39:18.636610 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:18.636239    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2" event={"ID":"88d5d6da-bc99-487e-959d-0f05e4525d27","Type":"ContainerDied","Data":"cc9452927557f9cc81574af03549bc0b43904f480d79a4cc91050994e0d04152"}
Jul 17 22:39:18.636610 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:18.636402    2565 scope.go:117] "RemoveContainer" containerID="cc9452927557f9cc81574af03549bc0b43904f480d79a4cc91050994e0d04152"
Jul 17 22:39:19.643443 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:19.643347    2565 generic.go:356] "Generic (PLEG): container finished" podID="88d5d6da-bc99-487e-959d-0f05e4525d27" containerID="aa9c61c7abc539c97d52642c5a24c7d673c6c4f314c06c857992dfffc3038eda" exitCode=6
Jul 17 22:39:19.643443 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:19.643415    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2" event={"ID":"88d5d6da-bc99-487e-959d-0f05e4525d27","Type":"ContainerDied","Data":"aa9c61c7abc539c97d52642c5a24c7d673c6c4f314c06c857992dfffc3038eda"}
Jul 17 22:39:19.643883 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:19.643468    2565 scope.go:117] "RemoveContainer" containerID="cc9452927557f9cc81574af03549bc0b43904f480d79a4cc91050994e0d04152"
Jul 17 22:39:19.643883 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:19.643744    2565 scope.go:117] "RemoveContainer" containerID="aa9c61c7abc539c97d52642c5a24c7d673c6c4f314c06c857992dfffc3038eda"
Jul 17 22:39:19.644007 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:39:19.643983    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2_odh-ai-gateway-infra(88d5d6da-bc99-487e-959d-0f05e4525d27)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2" podUID="88d5d6da-bc99-487e-959d-0f05e4525d27"
Jul 17 22:39:20.651581 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:20.651548    2565 scope.go:117] "RemoveContainer" containerID="aa9c61c7abc539c97d52642c5a24c7d673c6c4f314c06c857992dfffc3038eda"
Jul 17 22:39:20.651974 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:39:20.651775    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2_odh-ai-gateway-infra(88d5d6da-bc99-487e-959d-0f05e4525d27)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2" podUID="88d5d6da-bc99-487e-959d-0f05e4525d27"
Jul 17 22:39:25.355424 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:25.355389    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8"]
Jul 17 22:39:25.355815 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:25.355779    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45743089-59fb-449e-b289-771bb6ed5705" containerName="revoke-keys"
Jul 17 22:39:25.355815 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:25.355789    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="45743089-59fb-449e-b289-771bb6ed5705" containerName="revoke-keys"
Jul 17 22:39:25.355815 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:25.355805    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49f09421-593e-4a5e-9744-3cbb8a2ea7e7" containerName="revoke-keys"
Jul 17 22:39:25.355815 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:25.355810    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="49f09421-593e-4a5e-9744-3cbb8a2ea7e7" containerName="revoke-keys"
Jul 17 22:39:25.355949 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:25.355877    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="49f09421-593e-4a5e-9744-3cbb8a2ea7e7" containerName="revoke-keys"
Jul 17 22:39:25.355949 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:25.355886    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="45743089-59fb-449e-b289-771bb6ed5705" containerName="revoke-keys"
Jul 17 22:39:25.360301 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:25.360272    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8"
Jul 17 22:39:25.375132 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:25.375100    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8"]
Jul 17 22:39:25.443322 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:25.443277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66780f84-a16b-4a73-a336-fc3020397651-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8\" (UID: \"66780f84-a16b-4a73-a336-fc3020397651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8"
Jul 17 22:39:25.544209 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:25.544168    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66780f84-a16b-4a73-a336-fc3020397651-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8\" (UID: \"66780f84-a16b-4a73-a336-fc3020397651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8"
Jul 17 22:39:25.544791 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:25.544768    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66780f84-a16b-4a73-a336-fc3020397651-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8\" (UID: \"66780f84-a16b-4a73-a336-fc3020397651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8"
Jul 17 22:39:25.670276 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:25.670188    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8"
Jul 17 22:39:25.814682 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:25.814654    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8"]
Jul 17 22:39:25.816592 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:39:25.816557    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66780f84_a16b_4a73_a336_fc3020397651.slice/crio-e41474cefa751bda9d4c0a5d762e0671e410b17e3319c3aa7a874bf227dc51d0 WatchSource:0}: Error finding container e41474cefa751bda9d4c0a5d762e0671e410b17e3319c3aa7a874bf227dc51d0: Status 404 returned error can't find the container with id e41474cefa751bda9d4c0a5d762e0671e410b17e3319c3aa7a874bf227dc51d0
Jul 17 22:39:26.683440 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:26.683340    2565 generic.go:356] "Generic (PLEG): container finished" podID="66780f84-a16b-4a73-a336-fc3020397651" containerID="8ebd3cf9383d888c7a2258b73228773e334b79e9dbb894b15c5d8095a76da9ff" exitCode=6
Jul 17 22:39:26.683440 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:26.683384    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8" event={"ID":"66780f84-a16b-4a73-a336-fc3020397651","Type":"ContainerDied","Data":"8ebd3cf9383d888c7a2258b73228773e334b79e9dbb894b15c5d8095a76da9ff"}
Jul 17 22:39:26.683440 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:26.683429    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8" event={"ID":"66780f84-a16b-4a73-a336-fc3020397651","Type":"ContainerStarted","Data":"e41474cefa751bda9d4c0a5d762e0671e410b17e3319c3aa7a874bf227dc51d0"}
Jul 17 22:39:26.683950 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:26.683581    2565 scope.go:117] "RemoveContainer" containerID="8ebd3cf9383d888c7a2258b73228773e334b79e9dbb894b15c5d8095a76da9ff"
Jul 17 22:39:27.689954 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:27.689871    2565 generic.go:356] "Generic (PLEG): container finished" podID="66780f84-a16b-4a73-a336-fc3020397651" containerID="135b59e6ebf20ab6d5f423be4ad26cfb3ea4c2e46ffe032597cbedac9b29adb6" exitCode=6
Jul 17 22:39:27.690469 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:27.689960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8" event={"ID":"66780f84-a16b-4a73-a336-fc3020397651","Type":"ContainerDied","Data":"135b59e6ebf20ab6d5f423be4ad26cfb3ea4c2e46ffe032597cbedac9b29adb6"}
Jul 17 22:39:27.690469 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:27.690029    2565 scope.go:117] "RemoveContainer" containerID="8ebd3cf9383d888c7a2258b73228773e334b79e9dbb894b15c5d8095a76da9ff"
Jul 17 22:39:27.690469 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:27.690336    2565 scope.go:117] "RemoveContainer" containerID="135b59e6ebf20ab6d5f423be4ad26cfb3ea4c2e46ffe032597cbedac9b29adb6"
Jul 17 22:39:27.690607 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:39:27.690558    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8_odh-ai-gateway-infra(66780f84-a16b-4a73-a336-fc3020397651)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8" podUID="66780f84-a16b-4a73-a336-fc3020397651"
Jul 17 22:39:28.697293 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:28.697263    2565 scope.go:117] "RemoveContainer" containerID="135b59e6ebf20ab6d5f423be4ad26cfb3ea4c2e46ffe032597cbedac9b29adb6"
Jul 17 22:39:28.697681 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:39:28.697484    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8_odh-ai-gateway-infra(66780f84-a16b-4a73-a336-fc3020397651)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8" podUID="66780f84-a16b-4a73-a336-fc3020397651"
Jul 17 22:39:33.176958 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:33.176928    2565 scope.go:117] "RemoveContainer" containerID="aa9c61c7abc539c97d52642c5a24c7d673c6c4f314c06c857992dfffc3038eda"
Jul 17 22:39:33.722738 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:33.722645    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2" event={"ID":"88d5d6da-bc99-487e-959d-0f05e4525d27","Type":"ContainerStarted","Data":"1b7de74a3f244902f6b855f75079e435428dcfb03d033e0195a0984eb792f87d"}
Jul 17 22:39:33.755606 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:33.755558    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2" podStartSLOduration=17.755542077 podStartE2EDuration="17.755542077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:39:33.753757852 +0000 UTC m=+3237.142774502" watchObservedRunningTime="2026-07-17 22:39:33.755542077 +0000 UTC m=+3237.144558727"
Jul 17 22:39:34.055738 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:34.055700    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:39:34.729703 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:34.729667    2565 generic.go:356] "Generic (PLEG): container finished" podID="88d5d6da-bc99-487e-959d-0f05e4525d27" containerID="1b7de74a3f244902f6b855f75079e435428dcfb03d033e0195a0984eb792f87d" exitCode=6
Jul 17 22:39:34.730241 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:34.729730    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2" event={"ID":"88d5d6da-bc99-487e-959d-0f05e4525d27","Type":"ContainerDied","Data":"1b7de74a3f244902f6b855f75079e435428dcfb03d033e0195a0984eb792f87d"}
Jul 17 22:39:34.730241 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:34.729776    2565 scope.go:117] "RemoveContainer" containerID="aa9c61c7abc539c97d52642c5a24c7d673c6c4f314c06c857992dfffc3038eda"
Jul 17 22:39:34.730241 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:34.730106    2565 scope.go:117] "RemoveContainer" containerID="1b7de74a3f244902f6b855f75079e435428dcfb03d033e0195a0984eb792f87d"
Jul 17 22:39:34.730424 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:39:34.730401    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2_odh-ai-gateway-infra(88d5d6da-bc99-487e-959d-0f05e4525d27)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2" podUID="88d5d6da-bc99-487e-959d-0f05e4525d27"
Jul 17 22:39:34.766141 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:34.766101    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2"]
Jul 17 22:39:35.878333 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:35.878307    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2"
Jul 17 22:39:36.051074 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:36.051035    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d5d6da-bc99-487e-959d-0f05e4525d27-maas-api-service-ca\") pod \"88d5d6da-bc99-487e-959d-0f05e4525d27\" (UID: \"88d5d6da-bc99-487e-959d-0f05e4525d27\") "
Jul 17 22:39:36.051458 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:36.051432    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d5d6da-bc99-487e-959d-0f05e4525d27-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88d5d6da-bc99-487e-959d-0f05e4525d27" (UID: "88d5d6da-bc99-487e-959d-0f05e4525d27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:39:36.152633 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:36.152588    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d5d6da-bc99-487e-959d-0f05e4525d27-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:39:36.742887 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:36.742854    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2"
Jul 17 22:39:36.743091 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:36.742890    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2" event={"ID":"88d5d6da-bc99-487e-959d-0f05e4525d27","Type":"ContainerDied","Data":"c93b46c74d01d3fc8721454c0b1ff39e01e8f8b743bb863eec2923c31614cd03"}
Jul 17 22:39:36.743091 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:36.742956    2565 scope.go:117] "RemoveContainer" containerID="1b7de74a3f244902f6b855f75079e435428dcfb03d033e0195a0984eb792f87d"
Jul 17 22:39:36.765352 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:36.765323    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2"]
Jul 17 22:39:36.770596 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:36.770570    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-tjsl2"]
Jul 17 22:39:37.183086 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:37.183045    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88d5d6da-bc99-487e-959d-0f05e4525d27" path="/var/lib/kubelet/pods/88d5d6da-bc99-487e-959d-0f05e4525d27/volumes"
Jul 17 22:39:38.166476 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.166363    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:39:38.647307 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.647269    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:39:38.678994 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.678964    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn"]
Jul 17 22:39:38.679382 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.679368    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d5d6da-bc99-487e-959d-0f05e4525d27" containerName="revoke-keys"
Jul 17 22:39:38.679436 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.679384    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d5d6da-bc99-487e-959d-0f05e4525d27" containerName="revoke-keys"
Jul 17 22:39:38.679436 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.679396    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d5d6da-bc99-487e-959d-0f05e4525d27" containerName="revoke-keys"
Jul 17 22:39:38.679436 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.679402    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d5d6da-bc99-487e-959d-0f05e4525d27" containerName="revoke-keys"
Jul 17 22:39:38.679436 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.679411    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d5d6da-bc99-487e-959d-0f05e4525d27" containerName="revoke-keys"
Jul 17 22:39:38.679436 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.679416    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d5d6da-bc99-487e-959d-0f05e4525d27" containerName="revoke-keys"
Jul 17 22:39:38.679588 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.679466    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d5d6da-bc99-487e-959d-0f05e4525d27" containerName="revoke-keys"
Jul 17 22:39:38.679588 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.679473    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d5d6da-bc99-487e-959d-0f05e4525d27" containerName="revoke-keys"
Jul 17 22:39:38.679588 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.679480    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d5d6da-bc99-487e-959d-0f05e4525d27" containerName="revoke-keys"
Jul 17 22:39:38.683755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.683733    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn"
Jul 17 22:39:38.691510 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.691477    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn"]
Jul 17 22:39:38.777122 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.777052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f689345f-5dac-4338-b8fb-1d0e62259941-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn\" (UID: \"f689345f-5dac-4338-b8fb-1d0e62259941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn"
Jul 17 22:39:38.878293 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.878255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f689345f-5dac-4338-b8fb-1d0e62259941-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn\" (UID: \"f689345f-5dac-4338-b8fb-1d0e62259941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn"
Jul 17 22:39:38.878856 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.878837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f689345f-5dac-4338-b8fb-1d0e62259941-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn\" (UID: \"f689345f-5dac-4338-b8fb-1d0e62259941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn"
Jul 17 22:39:38.996383 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:38.996358    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn"
Jul 17 22:39:39.137285 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:39.137260    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn"]
Jul 17 22:39:39.139147 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:39:39.139117    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf689345f_5dac_4338_b8fb_1d0e62259941.slice/crio-98eb750da2e1fd99e1486021b3744e79a4ff6feb85d9aef8cc1a82d8ec7fca3c WatchSource:0}: Error finding container 98eb750da2e1fd99e1486021b3744e79a4ff6feb85d9aef8cc1a82d8ec7fca3c: Status 404 returned error can't find the container with id 98eb750da2e1fd99e1486021b3744e79a4ff6feb85d9aef8cc1a82d8ec7fca3c
Jul 17 22:39:39.760588 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:39.760555    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn" event={"ID":"f689345f-5dac-4338-b8fb-1d0e62259941","Type":"ContainerStarted","Data":"bc1db9ed4ee80c412c79a04eebcd27cb53af6028f4ebc2e46c40fc7fe0190a6a"}
Jul 17 22:39:39.760996 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:39.760610    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn" event={"ID":"f689345f-5dac-4338-b8fb-1d0e62259941","Type":"ContainerStarted","Data":"98eb750da2e1fd99e1486021b3744e79a4ff6feb85d9aef8cc1a82d8ec7fca3c"}
Jul 17 22:39:39.779144 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:39.779088    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn" podStartSLOduration=1.779045243 podStartE2EDuration="1.779045243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:39:39.777417337 +0000 UTC m=+3243.166433988" watchObservedRunningTime="2026-07-17 22:39:39.779045243 +0000 UTC m=+3243.168061894"
Jul 17 22:39:40.766976 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:40.766940    2565 generic.go:356] "Generic (PLEG): container finished" podID="f689345f-5dac-4338-b8fb-1d0e62259941" containerID="bc1db9ed4ee80c412c79a04eebcd27cb53af6028f4ebc2e46c40fc7fe0190a6a" exitCode=6
Jul 17 22:39:40.767411 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:40.767001    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn" event={"ID":"f689345f-5dac-4338-b8fb-1d0e62259941","Type":"ContainerDied","Data":"bc1db9ed4ee80c412c79a04eebcd27cb53af6028f4ebc2e46c40fc7fe0190a6a"}
Jul 17 22:39:40.767411 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:40.767287    2565 scope.go:117] "RemoveContainer" containerID="bc1db9ed4ee80c412c79a04eebcd27cb53af6028f4ebc2e46c40fc7fe0190a6a"
Jul 17 22:39:41.774161 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:41.774120    2565 generic.go:356] "Generic (PLEG): container finished" podID="f689345f-5dac-4338-b8fb-1d0e62259941" containerID="ac54e0795b7ddc439a123b1386ee72c4a802b3f61f18b0f84569d539d7105436" exitCode=6
Jul 17 22:39:41.774654 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:41.774214    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn" event={"ID":"f689345f-5dac-4338-b8fb-1d0e62259941","Type":"ContainerDied","Data":"ac54e0795b7ddc439a123b1386ee72c4a802b3f61f18b0f84569d539d7105436"}
Jul 17 22:39:41.774654 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:41.774274    2565 scope.go:117] "RemoveContainer" containerID="bc1db9ed4ee80c412c79a04eebcd27cb53af6028f4ebc2e46c40fc7fe0190a6a"
Jul 17 22:39:41.774654 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:41.774497    2565 scope.go:117] "RemoveContainer" containerID="ac54e0795b7ddc439a123b1386ee72c4a802b3f61f18b0f84569d539d7105436"
Jul 17 22:39:41.774777 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:39:41.774726    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn_odh-ai-gateway-infra(f689345f-5dac-4338-b8fb-1d0e62259941)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn" podUID="f689345f-5dac-4338-b8fb-1d0e62259941"
Jul 17 22:39:42.176547 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:42.176465    2565 scope.go:117] "RemoveContainer" containerID="135b59e6ebf20ab6d5f423be4ad26cfb3ea4c2e46ffe032597cbedac9b29adb6"
Jul 17 22:39:42.781140 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:42.781112    2565 scope.go:117] "RemoveContainer" containerID="ac54e0795b7ddc439a123b1386ee72c4a802b3f61f18b0f84569d539d7105436"
Jul 17 22:39:42.781554 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:39:42.781409    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn_odh-ai-gateway-infra(f689345f-5dac-4338-b8fb-1d0e62259941)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn" podUID="f689345f-5dac-4338-b8fb-1d0e62259941"
Jul 17 22:39:42.782621 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:42.782598    2565 generic.go:356] "Generic (PLEG): container finished" podID="66780f84-a16b-4a73-a336-fc3020397651" containerID="f44ddd45d55af70fbbea890ca8926935fbad4d5ee35872f2305de8205b8520ba" exitCode=6
Jul 17 22:39:42.782749 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:42.782671    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8" event={"ID":"66780f84-a16b-4a73-a336-fc3020397651","Type":"ContainerDied","Data":"f44ddd45d55af70fbbea890ca8926935fbad4d5ee35872f2305de8205b8520ba"}
Jul 17 22:39:42.782749 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:42.782736    2565 scope.go:117] "RemoveContainer" containerID="135b59e6ebf20ab6d5f423be4ad26cfb3ea4c2e46ffe032597cbedac9b29adb6"
Jul 17 22:39:42.783083 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:42.783047    2565 scope.go:117] "RemoveContainer" containerID="f44ddd45d55af70fbbea890ca8926935fbad4d5ee35872f2305de8205b8520ba"
Jul 17 22:39:42.783275 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:39:42.783254    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8_odh-ai-gateway-infra(66780f84-a16b-4a73-a336-fc3020397651)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8" podUID="66780f84-a16b-4a73-a336-fc3020397651"
Jul 17 22:39:43.281356 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:43.281323    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679"]
Jul 17 22:39:43.286214 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:43.286194    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679"
Jul 17 22:39:43.293136 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:43.293110    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679"]
Jul 17 22:39:43.314087 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:43.314036    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-rv679\" (UID: \"dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679"
Jul 17 22:39:43.415002 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:43.414960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-rv679\" (UID: \"dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679"
Jul 17 22:39:43.415618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:43.415595    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-rv679\" (UID: \"dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679"
Jul 17 22:39:43.597787 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:43.597701    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679"
Jul 17 22:39:43.736784 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:43.736759    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679"]
Jul 17 22:39:43.738835 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:39:43.738808    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf99ba5_eb7d_4881_bdc9_f4f16a614d8e.slice/crio-73e5dd3f7ce4b6eeec3a69475b018e70454abab7706ca018a16328948e27f7d6 WatchSource:0}: Error finding container 73e5dd3f7ce4b6eeec3a69475b018e70454abab7706ca018a16328948e27f7d6: Status 404 returned error can't find the container with id 73e5dd3f7ce4b6eeec3a69475b018e70454abab7706ca018a16328948e27f7d6
Jul 17 22:39:43.788216 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:43.788185    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679" event={"ID":"dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e","Type":"ContainerStarted","Data":"73e5dd3f7ce4b6eeec3a69475b018e70454abab7706ca018a16328948e27f7d6"}
Jul 17 22:39:43.832946 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:43.832912    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8"]
Jul 17 22:39:43.987135 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:43.987111    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8"
Jul 17 22:39:44.020890 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:44.020862    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66780f84-a16b-4a73-a336-fc3020397651-maas-api-service-ca\") pod \"66780f84-a16b-4a73-a336-fc3020397651\" (UID: \"66780f84-a16b-4a73-a336-fc3020397651\") "
Jul 17 22:39:44.021225 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:44.021204    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66780f84-a16b-4a73-a336-fc3020397651-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "66780f84-a16b-4a73-a336-fc3020397651" (UID: "66780f84-a16b-4a73-a336-fc3020397651"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:39:44.122330 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:44.122296    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66780f84-a16b-4a73-a336-fc3020397651-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:39:44.796276 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:44.796246    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8"
Jul 17 22:39:44.796737 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:44.796289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8" event={"ID":"66780f84-a16b-4a73-a336-fc3020397651","Type":"ContainerDied","Data":"e41474cefa751bda9d4c0a5d762e0671e410b17e3319c3aa7a874bf227dc51d0"}
Jul 17 22:39:44.796737 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:44.796356    2565 scope.go:117] "RemoveContainer" containerID="f44ddd45d55af70fbbea890ca8926935fbad4d5ee35872f2305de8205b8520ba"
Jul 17 22:39:44.797921 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:44.797895    2565 generic.go:356] "Generic (PLEG): container finished" podID="dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e" containerID="60131c0e45403b331a5b329260cc5a8c522683a8429688d9114a8e45d2f3bcb3" exitCode=6
Jul 17 22:39:44.798028 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:44.797922    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679" event={"ID":"dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e","Type":"ContainerDied","Data":"60131c0e45403b331a5b329260cc5a8c522683a8429688d9114a8e45d2f3bcb3"}
Jul 17 22:39:44.798226 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:44.798209    2565 scope.go:117] "RemoveContainer" containerID="60131c0e45403b331a5b329260cc5a8c522683a8429688d9114a8e45d2f3bcb3"
Jul 17 22:39:44.826572 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:44.826538    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8"]
Jul 17 22:39:44.830300 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:44.830274    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-9jkb8"]
Jul 17 22:39:45.182216 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:45.182135    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66780f84-a16b-4a73-a336-fc3020397651" path="/var/lib/kubelet/pods/66780f84-a16b-4a73-a336-fc3020397651/volumes"
Jul 17 22:39:45.804735 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:45.804691    2565 generic.go:356] "Generic (PLEG): container finished" podID="dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e" containerID="0e091c57c1a922ab7a82c50635908c018c05767814747b1c59393c7ac2cc0de4" exitCode=6
Jul 17 22:39:45.805220 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:45.804770    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679" event={"ID":"dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e","Type":"ContainerDied","Data":"0e091c57c1a922ab7a82c50635908c018c05767814747b1c59393c7ac2cc0de4"}
Jul 17 22:39:45.805220 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:45.804864    2565 scope.go:117] "RemoveContainer" containerID="60131c0e45403b331a5b329260cc5a8c522683a8429688d9114a8e45d2f3bcb3"
Jul 17 22:39:45.805220 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:45.805184    2565 scope.go:117] "RemoveContainer" containerID="0e091c57c1a922ab7a82c50635908c018c05767814747b1c59393c7ac2cc0de4"
Jul 17 22:39:45.805488 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:39:45.805458    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-rv679_odh-ai-gateway-infra(dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679" podUID="dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e"
Jul 17 22:39:46.812248 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:46.812221    2565 scope.go:117] "RemoveContainer" containerID="0e091c57c1a922ab7a82c50635908c018c05767814747b1c59393c7ac2cc0de4"
Jul 17 22:39:46.812641 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:39:46.812433    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-rv679_odh-ai-gateway-infra(dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679" podUID="dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e"
Jul 17 22:39:51.386773 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:51.386687    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:39:51.462866 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:51.462830    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:39:52.769509 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:52.769478    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:39:57.179380 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:57.179350    2565 scope.go:117] "RemoveContainer" containerID="0e091c57c1a922ab7a82c50635908c018c05767814747b1c59393c7ac2cc0de4"
Jul 17 22:39:57.179899 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:57.179465    2565 scope.go:117] "RemoveContainer" containerID="ac54e0795b7ddc439a123b1386ee72c4a802b3f61f18b0f84569d539d7105436"
Jul 17 22:39:57.869480 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:57.869438    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn" event={"ID":"f689345f-5dac-4338-b8fb-1d0e62259941","Type":"ContainerStarted","Data":"403be11fee6b25da869301aac645136dc07df5a6024aca3b658c470dbae4ce11"}
Jul 17 22:39:57.871355 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:57.871325    2565 generic.go:356] "Generic (PLEG): container finished" podID="dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e" containerID="41128d794ad0a123405646106665135cb6e80117efa1d00a03d207829033e0c0" exitCode=6
Jul 17 22:39:57.871482 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:57.871412    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679" event={"ID":"dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e","Type":"ContainerDied","Data":"41128d794ad0a123405646106665135cb6e80117efa1d00a03d207829033e0c0"}
Jul 17 22:39:57.871482 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:57.871473    2565 scope.go:117] "RemoveContainer" containerID="0e091c57c1a922ab7a82c50635908c018c05767814747b1c59393c7ac2cc0de4"
Jul 17 22:39:57.871804 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:57.871781    2565 scope.go:117] "RemoveContainer" containerID="41128d794ad0a123405646106665135cb6e80117efa1d00a03d207829033e0c0"
Jul 17 22:39:57.872026 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:39:57.872006    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-rv679_odh-ai-gateway-infra(dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679" podUID="dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e"
Jul 17 22:39:58.879527 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:58.879491    2565 generic.go:356] "Generic (PLEG): container finished" podID="f689345f-5dac-4338-b8fb-1d0e62259941" containerID="403be11fee6b25da869301aac645136dc07df5a6024aca3b658c470dbae4ce11" exitCode=6
Jul 17 22:39:58.880013 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:58.879570    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn" event={"ID":"f689345f-5dac-4338-b8fb-1d0e62259941","Type":"ContainerDied","Data":"403be11fee6b25da869301aac645136dc07df5a6024aca3b658c470dbae4ce11"}
Jul 17 22:39:58.880013 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:58.879631    2565 scope.go:117] "RemoveContainer" containerID="ac54e0795b7ddc439a123b1386ee72c4a802b3f61f18b0f84569d539d7105436"
Jul 17 22:39:58.880013 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:58.879953    2565 scope.go:117] "RemoveContainer" containerID="403be11fee6b25da869301aac645136dc07df5a6024aca3b658c470dbae4ce11"
Jul 17 22:39:58.880251 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:39:58.880227    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn_odh-ai-gateway-infra(f689345f-5dac-4338-b8fb-1d0e62259941)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn" podUID="f689345f-5dac-4338-b8fb-1d0e62259941"
Jul 17 22:39:58.914964 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:58.914929    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn"]
Jul 17 22:39:58.948374 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:58.948346    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679"]
Jul 17 22:39:59.096820 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:59.096791    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679"
Jul 17 22:39:59.171596 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:59.171495    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e-maas-api-service-ca\") pod \"dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e\" (UID: \"dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e\") "
Jul 17 22:39:59.171870 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:59.171845    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e" (UID: "dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:39:59.171981 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:59.171967    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:39:59.897337 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:59.897301    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679" event={"ID":"dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e","Type":"ContainerDied","Data":"73e5dd3f7ce4b6eeec3a69475b018e70454abab7706ca018a16328948e27f7d6"}
Jul 17 22:39:59.897929 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:59.897907    2565 scope.go:117] "RemoveContainer" containerID="41128d794ad0a123405646106665135cb6e80117efa1d00a03d207829033e0c0"
Jul 17 22:39:59.898083 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:59.898029    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679"
Jul 17 22:39:59.931214 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:59.931175    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679"]
Jul 17 22:39:59.937671 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:39:59.937644    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-rv679"]
Jul 17 22:40:00.040009 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:00.039982    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn"
Jul 17 22:40:00.182423 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:00.182334    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f689345f-5dac-4338-b8fb-1d0e62259941-maas-api-service-ca\") pod \"f689345f-5dac-4338-b8fb-1d0e62259941\" (UID: \"f689345f-5dac-4338-b8fb-1d0e62259941\") "
Jul 17 22:40:00.182723 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:00.182704    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f689345f-5dac-4338-b8fb-1d0e62259941-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f689345f-5dac-4338-b8fb-1d0e62259941" (UID: "f689345f-5dac-4338-b8fb-1d0e62259941"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:40:00.283665 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:00.283627    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f689345f-5dac-4338-b8fb-1d0e62259941-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:40:00.905558 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:00.905527    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn"
Jul 17 22:40:00.906028 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:00.905526    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn" event={"ID":"f689345f-5dac-4338-b8fb-1d0e62259941","Type":"ContainerDied","Data":"98eb750da2e1fd99e1486021b3744e79a4ff6feb85d9aef8cc1a82d8ec7fca3c"}
Jul 17 22:40:00.906028 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:00.905684    2565 scope.go:117] "RemoveContainer" containerID="403be11fee6b25da869301aac645136dc07df5a6024aca3b658c470dbae4ce11"
Jul 17 22:40:00.934484 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:00.934448    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn"]
Jul 17 22:40:00.940201 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:00.940164    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-p2qvn"]
Jul 17 22:40:01.181802 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:01.181724    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e" path="/var/lib/kubelet/pods/dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e/volumes"
Jul 17 22:40:01.182131 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:01.182117    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f689345f-5dac-4338-b8fb-1d0e62259941" path="/var/lib/kubelet/pods/f689345f-5dac-4338-b8fb-1d0e62259941/volumes"
Jul 17 22:40:11.957391 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:11.957354    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:40:12.048028 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:12.047990    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:40:15.449934 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.449885    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm"]
Jul 17 22:40:15.450470 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450443    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e" containerName="revoke-keys"
Jul 17 22:40:15.450470 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450463    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e" containerName="revoke-keys"
Jul 17 22:40:15.450592 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450479    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f689345f-5dac-4338-b8fb-1d0e62259941" containerName="revoke-keys"
Jul 17 22:40:15.450592 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450488    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f689345f-5dac-4338-b8fb-1d0e62259941" containerName="revoke-keys"
Jul 17 22:40:15.450592 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450499    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f689345f-5dac-4338-b8fb-1d0e62259941" containerName="revoke-keys"
Jul 17 22:40:15.450592 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450508    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f689345f-5dac-4338-b8fb-1d0e62259941" containerName="revoke-keys"
Jul 17 22:40:15.450592 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450532    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66780f84-a16b-4a73-a336-fc3020397651" containerName="revoke-keys"
Jul 17 22:40:15.450592 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450541    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66780f84-a16b-4a73-a336-fc3020397651" containerName="revoke-keys"
Jul 17 22:40:15.450592 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450556    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66780f84-a16b-4a73-a336-fc3020397651" containerName="revoke-keys"
Jul 17 22:40:15.450592 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450565    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66780f84-a16b-4a73-a336-fc3020397651" containerName="revoke-keys"
Jul 17 22:40:15.450592 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450573    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e" containerName="revoke-keys"
Jul 17 22:40:15.450592 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450581    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e" containerName="revoke-keys"
Jul 17 22:40:15.451095 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450682    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f689345f-5dac-4338-b8fb-1d0e62259941" containerName="revoke-keys"
Jul 17 22:40:15.451095 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450695    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="66780f84-a16b-4a73-a336-fc3020397651" containerName="revoke-keys"
Jul 17 22:40:15.451095 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450704    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e" containerName="revoke-keys"
Jul 17 22:40:15.451095 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450713    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="66780f84-a16b-4a73-a336-fc3020397651" containerName="revoke-keys"
Jul 17 22:40:15.451095 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450722    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e" containerName="revoke-keys"
Jul 17 22:40:15.451095 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450731    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e" containerName="revoke-keys"
Jul 17 22:40:15.451095 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450739    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="66780f84-a16b-4a73-a336-fc3020397651" containerName="revoke-keys"
Jul 17 22:40:15.451095 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.450750    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f689345f-5dac-4338-b8fb-1d0e62259941" containerName="revoke-keys"
Jul 17 22:40:15.455266 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.455242    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm"
Jul 17 22:40:15.462031 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.462002    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:40:15.462215 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.462016    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:40:15.477790 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.477753    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm"]
Jul 17 22:40:15.538725 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.538689    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:40:15.639283 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.639237    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e3ec8ad-0cf2-48e9-b7ea-2764866461a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm\" (UID: \"0e3ec8ad-0cf2-48e9-b7ea-2764866461a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm"
Jul 17 22:40:15.740523 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.740417    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e3ec8ad-0cf2-48e9-b7ea-2764866461a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm\" (UID: \"0e3ec8ad-0cf2-48e9-b7ea-2764866461a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm"
Jul 17 22:40:15.741042 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.741023    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e3ec8ad-0cf2-48e9-b7ea-2764866461a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm\" (UID: \"0e3ec8ad-0cf2-48e9-b7ea-2764866461a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm"
Jul 17 22:40:15.766458 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:15.766424    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm"
Jul 17 22:40:16.123650 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:16.123621    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm"]
Jul 17 22:40:16.124469 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:40:16.124442    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3ec8ad_0cf2_48e9_b7ea_2764866461a1.slice/crio-7dbdd16c210c315d78290b138dcb71a73f64843ce6c3ad6bd0176bda5425ae80 WatchSource:0}: Error finding container 7dbdd16c210c315d78290b138dcb71a73f64843ce6c3ad6bd0176bda5425ae80: Status 404 returned error can't find the container with id 7dbdd16c210c315d78290b138dcb71a73f64843ce6c3ad6bd0176bda5425ae80
Jul 17 22:40:16.994079 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:16.994033    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m"]
Jul 17 22:40:16.994530 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:16.994471    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f689345f-5dac-4338-b8fb-1d0e62259941" containerName="revoke-keys"
Jul 17 22:40:16.994530 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:16.994485    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f689345f-5dac-4338-b8fb-1d0e62259941" containerName="revoke-keys"
Jul 17 22:40:16.994530 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:16.994505    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66780f84-a16b-4a73-a336-fc3020397651" containerName="revoke-keys"
Jul 17 22:40:16.994530 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:16.994512    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66780f84-a16b-4a73-a336-fc3020397651" containerName="revoke-keys"
Jul 17 22:40:16.994530 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:16.994518    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e" containerName="revoke-keys"
Jul 17 22:40:16.994530 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:16.994524    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf99ba5-eb7d-4881-bdc9-f4f16a614d8e" containerName="revoke-keys"
Jul 17 22:40:16.994790 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:16.994580    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f689345f-5dac-4338-b8fb-1d0e62259941" containerName="revoke-keys"
Jul 17 22:40:16.997038 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:16.997007    2565 generic.go:356] "Generic (PLEG): container finished" podID="0e3ec8ad-0cf2-48e9-b7ea-2764866461a1" containerID="d5a0609fa2bc7ddf85c67fe5f4236c75234062183e29575aa04aafc8c743253f" exitCode=6
Jul 17 22:40:16.997414 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:16.997397    2565 scope.go:117] "RemoveContainer" containerID="d5a0609fa2bc7ddf85c67fe5f4236c75234062183e29575aa04aafc8c743253f"
Jul 17 22:40:16.997614 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:16.997593    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm" event={"ID":"0e3ec8ad-0cf2-48e9-b7ea-2764866461a1","Type":"ContainerDied","Data":"d5a0609fa2bc7ddf85c67fe5f4236c75234062183e29575aa04aafc8c743253f"}
Jul 17 22:40:16.997699 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:16.997624    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm" event={"ID":"0e3ec8ad-0cf2-48e9-b7ea-2764866461a1","Type":"ContainerStarted","Data":"7dbdd16c210c315d78290b138dcb71a73f64843ce6c3ad6bd0176bda5425ae80"}
Jul 17 22:40:16.997699 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:16.997689    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m"
Jul 17 22:40:17.024057 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:17.024028    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m"]
Jul 17 22:40:17.155828 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:17.155780    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab765028-f13d-4b75-8a4f-26b2ce6d2af5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m\" (UID: \"ab765028-f13d-4b75-8a4f-26b2ce6d2af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m"
Jul 17 22:40:17.256748 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:17.256707    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab765028-f13d-4b75-8a4f-26b2ce6d2af5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m\" (UID: \"ab765028-f13d-4b75-8a4f-26b2ce6d2af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m"
Jul 17 22:40:17.257577 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:17.257537    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab765028-f13d-4b75-8a4f-26b2ce6d2af5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m\" (UID: \"ab765028-f13d-4b75-8a4f-26b2ce6d2af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m"
Jul 17 22:40:17.308710 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:17.308670    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m"
Jul 17 22:40:17.464091 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:17.464031    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m"]
Jul 17 22:40:17.466035 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:40:17.465999    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab765028_f13d_4b75_8a4f_26b2ce6d2af5.slice/crio-b1ee2d0cb876e97b307f63801b9e8fe40871efdb774eb2e8e09add0d200ba54c WatchSource:0}: Error finding container b1ee2d0cb876e97b307f63801b9e8fe40871efdb774eb2e8e09add0d200ba54c: Status 404 returned error can't find the container with id b1ee2d0cb876e97b307f63801b9e8fe40871efdb774eb2e8e09add0d200ba54c
Jul 17 22:40:18.004820 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:18.004784    2565 generic.go:356] "Generic (PLEG): container finished" podID="0e3ec8ad-0cf2-48e9-b7ea-2764866461a1" containerID="69f21101504e16429833d5ca5541d1fb734cb4ee3b25e5846a41da175ad9f6a0" exitCode=6
Jul 17 22:40:18.005329 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:18.004846    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm" event={"ID":"0e3ec8ad-0cf2-48e9-b7ea-2764866461a1","Type":"ContainerDied","Data":"69f21101504e16429833d5ca5541d1fb734cb4ee3b25e5846a41da175ad9f6a0"}
Jul 17 22:40:18.005329 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:18.004904    2565 scope.go:117] "RemoveContainer" containerID="d5a0609fa2bc7ddf85c67fe5f4236c75234062183e29575aa04aafc8c743253f"
Jul 17 22:40:18.005329 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:18.005112    2565 scope.go:117] "RemoveContainer" containerID="69f21101504e16429833d5ca5541d1fb734cb4ee3b25e5846a41da175ad9f6a0"
Jul 17 22:40:18.005485 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:40:18.005345    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm_odh-ai-gateway-infra(0e3ec8ad-0cf2-48e9-b7ea-2764866461a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm" podUID="0e3ec8ad-0cf2-48e9-b7ea-2764866461a1"
Jul 17 22:40:18.006503 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:18.006477    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m" event={"ID":"ab765028-f13d-4b75-8a4f-26b2ce6d2af5","Type":"ContainerStarted","Data":"d6bf699e717be6e7caf64d684f255fa33742153e91bc0ce6bcbfabf6ca177116"}
Jul 17 22:40:18.006605 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:18.006522    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m" event={"ID":"ab765028-f13d-4b75-8a4f-26b2ce6d2af5","Type":"ContainerStarted","Data":"b1ee2d0cb876e97b307f63801b9e8fe40871efdb774eb2e8e09add0d200ba54c"}
Jul 17 22:40:18.043333 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:18.043272    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m" podStartSLOduration=2.043255349 podStartE2EDuration="2.043255349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:40:18.040263045 +0000 UTC m=+3281.429279696" watchObservedRunningTime="2026-07-17 22:40:18.043255349 +0000 UTC m=+3281.432271999"
Jul 17 22:40:19.013488 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:19.013462    2565 scope.go:117] "RemoveContainer" containerID="69f21101504e16429833d5ca5541d1fb734cb4ee3b25e5846a41da175ad9f6a0"
Jul 17 22:40:19.013948 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:40:19.013749    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm_odh-ai-gateway-infra(0e3ec8ad-0cf2-48e9-b7ea-2764866461a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm" podUID="0e3ec8ad-0cf2-48e9-b7ea-2764866461a1"
Jul 17 22:40:19.014858 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:19.014832    2565 generic.go:356] "Generic (PLEG): container finished" podID="ab765028-f13d-4b75-8a4f-26b2ce6d2af5" containerID="d6bf699e717be6e7caf64d684f255fa33742153e91bc0ce6bcbfabf6ca177116" exitCode=6
Jul 17 22:40:19.014978 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:19.014917    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m" event={"ID":"ab765028-f13d-4b75-8a4f-26b2ce6d2af5","Type":"ContainerDied","Data":"d6bf699e717be6e7caf64d684f255fa33742153e91bc0ce6bcbfabf6ca177116"}
Jul 17 22:40:19.015109 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:19.015097    2565 scope.go:117] "RemoveContainer" containerID="d6bf699e717be6e7caf64d684f255fa33742153e91bc0ce6bcbfabf6ca177116"
Jul 17 22:40:20.021528 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:20.021492    2565 generic.go:356] "Generic (PLEG): container finished" podID="ab765028-f13d-4b75-8a4f-26b2ce6d2af5" containerID="db31e9f688ce527d923274f61057af8f63d16e7a7658335f6dc7883843f4ffb8" exitCode=6
Jul 17 22:40:20.022006 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:20.021562    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m" event={"ID":"ab765028-f13d-4b75-8a4f-26b2ce6d2af5","Type":"ContainerDied","Data":"db31e9f688ce527d923274f61057af8f63d16e7a7658335f6dc7883843f4ffb8"}
Jul 17 22:40:20.022006 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:20.021609    2565 scope.go:117] "RemoveContainer" containerID="d6bf699e717be6e7caf64d684f255fa33742153e91bc0ce6bcbfabf6ca177116"
Jul 17 22:40:20.022006 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:20.021848    2565 scope.go:117] "RemoveContainer" containerID="db31e9f688ce527d923274f61057af8f63d16e7a7658335f6dc7883843f4ffb8"
Jul 17 22:40:20.022247 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:40:20.022155    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m_odh-ai-gateway-infra(ab765028-f13d-4b75-8a4f-26b2ce6d2af5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m" podUID="ab765028-f13d-4b75-8a4f-26b2ce6d2af5"
Jul 17 22:40:21.028942 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:21.028913    2565 scope.go:117] "RemoveContainer" containerID="db31e9f688ce527d923274f61057af8f63d16e7a7658335f6dc7883843f4ffb8"
Jul 17 22:40:21.029385 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:40:21.029185    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m_odh-ai-gateway-infra(ab765028-f13d-4b75-8a4f-26b2ce6d2af5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m" podUID="ab765028-f13d-4b75-8a4f-26b2ce6d2af5"
Jul 17 22:40:25.280724 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:25.280680    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:40:30.176758 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:30.176723    2565 scope.go:117] "RemoveContainer" containerID="69f21101504e16429833d5ca5541d1fb734cb4ee3b25e5846a41da175ad9f6a0"
Jul 17 22:40:31.080303 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:31.080268    2565 generic.go:356] "Generic (PLEG): container finished" podID="0e3ec8ad-0cf2-48e9-b7ea-2764866461a1" containerID="dd79c2930e4a37db7aec48c743a87b83bde398bdbde13b959dc42b1f879f1392" exitCode=6
Jul 17 22:40:31.080505 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:31.080322    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm" event={"ID":"0e3ec8ad-0cf2-48e9-b7ea-2764866461a1","Type":"ContainerDied","Data":"dd79c2930e4a37db7aec48c743a87b83bde398bdbde13b959dc42b1f879f1392"}
Jul 17 22:40:31.080505 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:31.080371    2565 scope.go:117] "RemoveContainer" containerID="69f21101504e16429833d5ca5541d1fb734cb4ee3b25e5846a41da175ad9f6a0"
Jul 17 22:40:31.080705 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:31.080680    2565 scope.go:117] "RemoveContainer" containerID="dd79c2930e4a37db7aec48c743a87b83bde398bdbde13b959dc42b1f879f1392"
Jul 17 22:40:31.080944 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:40:31.080924    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm_odh-ai-gateway-infra(0e3ec8ad-0cf2-48e9-b7ea-2764866461a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm" podUID="0e3ec8ad-0cf2-48e9-b7ea-2764866461a1"
Jul 17 22:40:31.366740 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:31.366700    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:40:32.131104 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:32.131047    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm"]
Jul 17 22:40:32.271098 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:32.271053    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm"
Jul 17 22:40:32.397730 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:32.397644    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e3ec8ad-0cf2-48e9-b7ea-2764866461a1-maas-api-service-ca\") pod \"0e3ec8ad-0cf2-48e9-b7ea-2764866461a1\" (UID: \"0e3ec8ad-0cf2-48e9-b7ea-2764866461a1\") "
Jul 17 22:40:32.398150 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:32.397993    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3ec8ad-0cf2-48e9-b7ea-2764866461a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e3ec8ad-0cf2-48e9-b7ea-2764866461a1" (UID: "0e3ec8ad-0cf2-48e9-b7ea-2764866461a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:40:32.498608 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:32.498563    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e3ec8ad-0cf2-48e9-b7ea-2764866461a1-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:40:33.093222 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.093186    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm" event={"ID":"0e3ec8ad-0cf2-48e9-b7ea-2764866461a1","Type":"ContainerDied","Data":"7dbdd16c210c315d78290b138dcb71a73f64843ce6c3ad6bd0176bda5425ae80"}
Jul 17 22:40:33.093222 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.093218    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm"
Jul 17 22:40:33.093455 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.093250    2565 scope.go:117] "RemoveContainer" containerID="dd79c2930e4a37db7aec48c743a87b83bde398bdbde13b959dc42b1f879f1392"
Jul 17 22:40:33.116360 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.116327    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm"]
Jul 17 22:40:33.121577 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.121550    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-xj8xm"]
Jul 17 22:40:33.182553 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.182522    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e3ec8ad-0cf2-48e9-b7ea-2764866461a1" path="/var/lib/kubelet/pods/0e3ec8ad-0cf2-48e9-b7ea-2764866461a1/volumes"
Jul 17 22:40:33.248602 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.248569    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:40:33.375567 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.375483    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh"]
Jul 17 22:40:33.375872 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.375859    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e3ec8ad-0cf2-48e9-b7ea-2764866461a1" containerName="revoke-keys"
Jul 17 22:40:33.375920 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.375873    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e3ec8ad-0cf2-48e9-b7ea-2764866461a1" containerName="revoke-keys"
Jul 17 22:40:33.375920 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.375884    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e3ec8ad-0cf2-48e9-b7ea-2764866461a1" containerName="revoke-keys"
Jul 17 22:40:33.375920 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.375889    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e3ec8ad-0cf2-48e9-b7ea-2764866461a1" containerName="revoke-keys"
Jul 17 22:40:33.375920 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.375898    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e3ec8ad-0cf2-48e9-b7ea-2764866461a1" containerName="revoke-keys"
Jul 17 22:40:33.375920 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.375904    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e3ec8ad-0cf2-48e9-b7ea-2764866461a1" containerName="revoke-keys"
Jul 17 22:40:33.376121 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.375961    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e3ec8ad-0cf2-48e9-b7ea-2764866461a1" containerName="revoke-keys"
Jul 17 22:40:33.376121 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.375967    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e3ec8ad-0cf2-48e9-b7ea-2764866461a1" containerName="revoke-keys"
Jul 17 22:40:33.380403 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.380385    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh"
Jul 17 22:40:33.393224 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.393195    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh"]
Jul 17 22:40:33.420318 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.420285    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:40:33.507202 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.507166    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95fb333d-9439-4f99-8116-87f7c3c20fe5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-5glwh\" (UID: \"95fb333d-9439-4f99-8116-87f7c3c20fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh"
Jul 17 22:40:33.607932 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.607894    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95fb333d-9439-4f99-8116-87f7c3c20fe5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-5glwh\" (UID: \"95fb333d-9439-4f99-8116-87f7c3c20fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh"
Jul 17 22:40:33.608512 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.608493    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95fb333d-9439-4f99-8116-87f7c3c20fe5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-5glwh\" (UID: \"95fb333d-9439-4f99-8116-87f7c3c20fe5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh"
Jul 17 22:40:33.690622 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.690528    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh"
Jul 17 22:40:33.828656 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:33.828625    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh"]
Jul 17 22:40:33.831039 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:40:33.831011    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95fb333d_9439_4f99_8116_87f7c3c20fe5.slice/crio-38cd1807f6e6974bbe94d48911c3db344bb88f2f969b2009b3a33a9c84092f5e WatchSource:0}: Error finding container 38cd1807f6e6974bbe94d48911c3db344bb88f2f969b2009b3a33a9c84092f5e: Status 404 returned error can't find the container with id 38cd1807f6e6974bbe94d48911c3db344bb88f2f969b2009b3a33a9c84092f5e
Jul 17 22:40:34.107020 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:34.106983    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh" event={"ID":"95fb333d-9439-4f99-8116-87f7c3c20fe5","Type":"ContainerStarted","Data":"38cd1807f6e6974bbe94d48911c3db344bb88f2f969b2009b3a33a9c84092f5e"}
Jul 17 22:40:35.113528 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:35.113492    2565 generic.go:356] "Generic (PLEG): container finished" podID="95fb333d-9439-4f99-8116-87f7c3c20fe5" containerID="aa7f7e66f5d6002fa1ed68acc70a29d7d017c9879a93dd18cc7446bcf3df0953" exitCode=6
Jul 17 22:40:35.113915 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:35.113579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh" event={"ID":"95fb333d-9439-4f99-8116-87f7c3c20fe5","Type":"ContainerDied","Data":"aa7f7e66f5d6002fa1ed68acc70a29d7d017c9879a93dd18cc7446bcf3df0953"}
Jul 17 22:40:35.113915 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:35.113722    2565 scope.go:117] "RemoveContainer" containerID="aa7f7e66f5d6002fa1ed68acc70a29d7d017c9879a93dd18cc7446bcf3df0953"
Jul 17 22:40:35.177151 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:35.177108    2565 scope.go:117] "RemoveContainer" containerID="db31e9f688ce527d923274f61057af8f63d16e7a7658335f6dc7883843f4ffb8"
Jul 17 22:40:36.120028 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:36.119940    2565 generic.go:356] "Generic (PLEG): container finished" podID="ab765028-f13d-4b75-8a4f-26b2ce6d2af5" containerID="d8aefc4da87b140f84a84a042ed4062931c2993092394712a1202066f7a5f6d6" exitCode=6
Jul 17 22:40:36.120028 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:36.120017    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m" event={"ID":"ab765028-f13d-4b75-8a4f-26b2ce6d2af5","Type":"ContainerDied","Data":"d8aefc4da87b140f84a84a042ed4062931c2993092394712a1202066f7a5f6d6"}
Jul 17 22:40:36.120661 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:36.120106    2565 scope.go:117] "RemoveContainer" containerID="db31e9f688ce527d923274f61057af8f63d16e7a7658335f6dc7883843f4ffb8"
Jul 17 22:40:36.120661 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:36.120500    2565 scope.go:117] "RemoveContainer" containerID="d8aefc4da87b140f84a84a042ed4062931c2993092394712a1202066f7a5f6d6"
Jul 17 22:40:36.120778 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:40:36.120753    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m_odh-ai-gateway-infra(ab765028-f13d-4b75-8a4f-26b2ce6d2af5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m" podUID="ab765028-f13d-4b75-8a4f-26b2ce6d2af5"
Jul 17 22:40:36.121910 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:36.121886    2565 generic.go:356] "Generic (PLEG): container finished" podID="95fb333d-9439-4f99-8116-87f7c3c20fe5" containerID="aa31b5738ed50201fef6d99376139c46c70786ca8e2d440b77a5cfc8a86dab41" exitCode=6
Jul 17 22:40:36.122003 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:36.121964    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh" event={"ID":"95fb333d-9439-4f99-8116-87f7c3c20fe5","Type":"ContainerDied","Data":"aa31b5738ed50201fef6d99376139c46c70786ca8e2d440b77a5cfc8a86dab41"}
Jul 17 22:40:36.122205 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:36.122192    2565 scope.go:117] "RemoveContainer" containerID="aa31b5738ed50201fef6d99376139c46c70786ca8e2d440b77a5cfc8a86dab41"
Jul 17 22:40:36.122379 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:40:36.122363    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-5glwh_odh-ai-gateway-infra(95fb333d-9439-4f99-8116-87f7c3c20fe5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh" podUID="95fb333d-9439-4f99-8116-87f7c3c20fe5"
Jul 17 22:40:36.131688 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:36.131669    2565 scope.go:117] "RemoveContainer" containerID="aa7f7e66f5d6002fa1ed68acc70a29d7d017c9879a93dd18cc7446bcf3df0953"
Jul 17 22:40:37.131194 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:37.131151    2565 scope.go:117] "RemoveContainer" containerID="aa31b5738ed50201fef6d99376139c46c70786ca8e2d440b77a5cfc8a86dab41"
Jul 17 22:40:37.131593 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:40:37.131406    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-5glwh_odh-ai-gateway-infra(95fb333d-9439-4f99-8116-87f7c3c20fe5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh" podUID="95fb333d-9439-4f99-8116-87f7c3c20fe5"
Jul 17 22:40:37.153397 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:37.153364    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m"]
Jul 17 22:40:37.302098 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:37.302037    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m"
Jul 17 22:40:37.343169 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:37.343136    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab765028-f13d-4b75-8a4f-26b2ce6d2af5-maas-api-service-ca\") pod \"ab765028-f13d-4b75-8a4f-26b2ce6d2af5\" (UID: \"ab765028-f13d-4b75-8a4f-26b2ce6d2af5\") "
Jul 17 22:40:37.343581 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:37.343545    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab765028-f13d-4b75-8a4f-26b2ce6d2af5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ab765028-f13d-4b75-8a4f-26b2ce6d2af5" (UID: "ab765028-f13d-4b75-8a4f-26b2ce6d2af5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:40:37.359519 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:37.359493    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:40:37.372581 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:37.372559    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bhftj_4ce379ff-d4b8-443c-8a1d-b522b7f9f5e6/ovn-acl-logging/0.log"
Jul 17 22:40:37.444832 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:37.444742    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab765028-f13d-4b75-8a4f-26b2ce6d2af5-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:40:38.139727 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.139628    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m" event={"ID":"ab765028-f13d-4b75-8a4f-26b2ce6d2af5","Type":"ContainerDied","Data":"b1ee2d0cb876e97b307f63801b9e8fe40871efdb774eb2e8e09add0d200ba54c"}
Jul 17 22:40:38.139727 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.139670    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m"
Jul 17 22:40:38.140251 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.139741    2565 scope.go:117] "RemoveContainer" containerID="d8aefc4da87b140f84a84a042ed4062931c2993092394712a1202066f7a5f6d6"
Jul 17 22:40:38.201714 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.201657    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m"]
Jul 17 22:40:38.203854 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.203831    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-mhx6m"]
Jul 17 22:40:38.733906 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.733875    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc"]
Jul 17 22:40:38.734280 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.734267    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab765028-f13d-4b75-8a4f-26b2ce6d2af5" containerName="revoke-keys"
Jul 17 22:40:38.734335 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.734282    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab765028-f13d-4b75-8a4f-26b2ce6d2af5" containerName="revoke-keys"
Jul 17 22:40:38.734335 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.734291    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab765028-f13d-4b75-8a4f-26b2ce6d2af5" containerName="revoke-keys"
Jul 17 22:40:38.734335 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.734296    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab765028-f13d-4b75-8a4f-26b2ce6d2af5" containerName="revoke-keys"
Jul 17 22:40:38.734430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.734362    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab765028-f13d-4b75-8a4f-26b2ce6d2af5" containerName="revoke-keys"
Jul 17 22:40:38.734430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.734373    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab765028-f13d-4b75-8a4f-26b2ce6d2af5" containerName="revoke-keys"
Jul 17 22:40:38.734430 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.734378    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e3ec8ad-0cf2-48e9-b7ea-2764866461a1" containerName="revoke-keys"
Jul 17 22:40:38.739038 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.739018    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc"
Jul 17 22:40:38.746300 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.746261    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc"]
Jul 17 22:40:38.858867 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.858825    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/894a47b3-9073-4ddf-9998-5e4ad8bea2a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc\" (UID: \"894a47b3-9073-4ddf-9998-5e4ad8bea2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc"
Jul 17 22:40:38.959836 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.959781    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/894a47b3-9073-4ddf-9998-5e4ad8bea2a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc\" (UID: \"894a47b3-9073-4ddf-9998-5e4ad8bea2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc"
Jul 17 22:40:38.960649 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:38.960621    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/894a47b3-9073-4ddf-9998-5e4ad8bea2a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc\" (UID: \"894a47b3-9073-4ddf-9998-5e4ad8bea2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc"
Jul 17 22:40:39.052419 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:39.052380    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc"
Jul 17 22:40:39.184037 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:39.183981    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab765028-f13d-4b75-8a4f-26b2ce6d2af5" path="/var/lib/kubelet/pods/ab765028-f13d-4b75-8a4f-26b2ce6d2af5/volumes"
Jul 17 22:40:39.199018 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:39.198984    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc"]
Jul 17 22:40:39.202330 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:40:39.202304    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894a47b3_9073_4ddf_9998_5e4ad8bea2a6.slice/crio-5fe6b3261c794ce6f1426831b3ca826acd7fd55370d889567d1eb0454433163f WatchSource:0}: Error finding container 5fe6b3261c794ce6f1426831b3ca826acd7fd55370d889567d1eb0454433163f: Status 404 returned error can't find the container with id 5fe6b3261c794ce6f1426831b3ca826acd7fd55370d889567d1eb0454433163f
Jul 17 22:40:40.154747 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:40.154658    2565 generic.go:356] "Generic (PLEG): container finished" podID="894a47b3-9073-4ddf-9998-5e4ad8bea2a6" containerID="24ee53567b18f10833f2c5823a0089bebb37f335f6ffcc5350067e1a3481b991" exitCode=6
Jul 17 22:40:40.154908 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:40.154747    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc" event={"ID":"894a47b3-9073-4ddf-9998-5e4ad8bea2a6","Type":"ContainerDied","Data":"24ee53567b18f10833f2c5823a0089bebb37f335f6ffcc5350067e1a3481b991"}
Jul 17 22:40:40.154908 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:40.154804    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc" event={"ID":"894a47b3-9073-4ddf-9998-5e4ad8bea2a6","Type":"ContainerStarted","Data":"5fe6b3261c794ce6f1426831b3ca826acd7fd55370d889567d1eb0454433163f"}
Jul 17 22:40:40.154982 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:40.154940    2565 scope.go:117] "RemoveContainer" containerID="24ee53567b18f10833f2c5823a0089bebb37f335f6ffcc5350067e1a3481b991"
Jul 17 22:40:41.162773 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:41.162677    2565 generic.go:356] "Generic (PLEG): container finished" podID="894a47b3-9073-4ddf-9998-5e4ad8bea2a6" containerID="307295c63f64c2fd97ec0b660d495466d1bd5f65747ff540fddee28d232f9745" exitCode=6
Jul 17 22:40:41.163299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:41.162766    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc" event={"ID":"894a47b3-9073-4ddf-9998-5e4ad8bea2a6","Type":"ContainerDied","Data":"307295c63f64c2fd97ec0b660d495466d1bd5f65747ff540fddee28d232f9745"}
Jul 17 22:40:41.163299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:41.162835    2565 scope.go:117] "RemoveContainer" containerID="24ee53567b18f10833f2c5823a0089bebb37f335f6ffcc5350067e1a3481b991"
Jul 17 22:40:41.163299 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:41.163061    2565 scope.go:117] "RemoveContainer" containerID="307295c63f64c2fd97ec0b660d495466d1bd5f65747ff540fddee28d232f9745"
Jul 17 22:40:41.163459 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:40:41.163340    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc_odh-ai-gateway-infra(894a47b3-9073-4ddf-9998-5e4ad8bea2a6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc" podUID="894a47b3-9073-4ddf-9998-5e4ad8bea2a6"
Jul 17 22:40:42.169552 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:42.169528    2565 scope.go:117] "RemoveContainer" containerID="307295c63f64c2fd97ec0b660d495466d1bd5f65747ff540fddee28d232f9745"
Jul 17 22:40:42.169984 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:40:42.169719    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc_odh-ai-gateway-infra(894a47b3-9073-4ddf-9998-5e4ad8bea2a6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc" podUID="894a47b3-9073-4ddf-9998-5e4ad8bea2a6"
Jul 17 22:40:48.176868 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:48.176832    2565 scope.go:117] "RemoveContainer" containerID="aa31b5738ed50201fef6d99376139c46c70786ca8e2d440b77a5cfc8a86dab41"
Jul 17 22:40:49.205490 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:49.205400    2565 generic.go:356] "Generic (PLEG): container finished" podID="95fb333d-9439-4f99-8116-87f7c3c20fe5" containerID="f58d311084e0a916514f36a128e28f4cb326fa475ce73b3e95438556e976e382" exitCode=6
Jul 17 22:40:49.205490 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:49.205472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh" event={"ID":"95fb333d-9439-4f99-8116-87f7c3c20fe5","Type":"ContainerDied","Data":"f58d311084e0a916514f36a128e28f4cb326fa475ce73b3e95438556e976e382"}
Jul 17 22:40:49.205961 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:49.205532    2565 scope.go:117] "RemoveContainer" containerID="aa31b5738ed50201fef6d99376139c46c70786ca8e2d440b77a5cfc8a86dab41"
Jul 17 22:40:49.205961 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:49.205773    2565 scope.go:117] "RemoveContainer" containerID="f58d311084e0a916514f36a128e28f4cb326fa475ce73b3e95438556e976e382"
Jul 17 22:40:49.206059 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:40:49.206003    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-5glwh_odh-ai-gateway-infra(95fb333d-9439-4f99-8116-87f7c3c20fe5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh" podUID="95fb333d-9439-4f99-8116-87f7c3c20fe5"
Jul 17 22:40:50.238227 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:50.238186    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh"]
Jul 17 22:40:50.366911 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:50.366878    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh"
Jul 17 22:40:50.478453 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:50.478406    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95fb333d-9439-4f99-8116-87f7c3c20fe5-maas-api-service-ca\") pod \"95fb333d-9439-4f99-8116-87f7c3c20fe5\" (UID: \"95fb333d-9439-4f99-8116-87f7c3c20fe5\") "
Jul 17 22:40:50.478790 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:50.478762    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95fb333d-9439-4f99-8116-87f7c3c20fe5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "95fb333d-9439-4f99-8116-87f7c3c20fe5" (UID: "95fb333d-9439-4f99-8116-87f7c3c20fe5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:40:50.580279 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:50.580240    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95fb333d-9439-4f99-8116-87f7c3c20fe5-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:40:51.219693 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:51.219658    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh"
Jul 17 22:40:51.219693 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:51.219689    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh" event={"ID":"95fb333d-9439-4f99-8116-87f7c3c20fe5","Type":"ContainerDied","Data":"38cd1807f6e6974bbe94d48911c3db344bb88f2f969b2009b3a33a9c84092f5e"}
Jul 17 22:40:51.219946 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:51.219748    2565 scope.go:117] "RemoveContainer" containerID="f58d311084e0a916514f36a128e28f4cb326fa475ce73b3e95438556e976e382"
Jul 17 22:40:51.238660 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:51.238627    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh"]
Jul 17 22:40:51.243255 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:51.243228    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-5glwh"]
Jul 17 22:40:53.182756 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:53.182715    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95fb333d-9439-4f99-8116-87f7c3c20fe5" path="/var/lib/kubelet/pods/95fb333d-9439-4f99-8116-87f7c3c20fe5/volumes"
Jul 17 22:40:54.177226 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:54.177195    2565 scope.go:117] "RemoveContainer" containerID="307295c63f64c2fd97ec0b660d495466d1bd5f65747ff540fddee28d232f9745"
Jul 17 22:40:55.242969 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:55.242882    2565 generic.go:356] "Generic (PLEG): container finished" podID="894a47b3-9073-4ddf-9998-5e4ad8bea2a6" containerID="5ada5896cb80bc91206b74f637c204f9d6d5a9ca3bd2204a335a905ec0666d53" exitCode=6
Jul 17 22:40:55.243447 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:55.242957    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc" event={"ID":"894a47b3-9073-4ddf-9998-5e4ad8bea2a6","Type":"ContainerDied","Data":"5ada5896cb80bc91206b74f637c204f9d6d5a9ca3bd2204a335a905ec0666d53"}
Jul 17 22:40:55.243447 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:55.243026    2565 scope.go:117] "RemoveContainer" containerID="307295c63f64c2fd97ec0b660d495466d1bd5f65747ff540fddee28d232f9745"
Jul 17 22:40:55.243447 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:55.243342    2565 scope.go:117] "RemoveContainer" containerID="5ada5896cb80bc91206b74f637c204f9d6d5a9ca3bd2204a335a905ec0666d53"
Jul 17 22:40:55.243645 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:40:55.243621    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc_odh-ai-gateway-infra(894a47b3-9073-4ddf-9998-5e4ad8bea2a6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc" podUID="894a47b3-9073-4ddf-9998-5e4ad8bea2a6"
Jul 17 22:40:56.273016 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:56.272982    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc"]
Jul 17 22:40:56.412261 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:56.412236    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc"
Jul 17 22:40:56.535308 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:56.535201    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/894a47b3-9073-4ddf-9998-5e4ad8bea2a6-maas-api-service-ca\") pod \"894a47b3-9073-4ddf-9998-5e4ad8bea2a6\" (UID: \"894a47b3-9073-4ddf-9998-5e4ad8bea2a6\") "
Jul 17 22:40:56.535587 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:56.535560    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894a47b3-9073-4ddf-9998-5e4ad8bea2a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "894a47b3-9073-4ddf-9998-5e4ad8bea2a6" (UID: "894a47b3-9073-4ddf-9998-5e4ad8bea2a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:40:56.636898 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:56.636847    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/894a47b3-9073-4ddf-9998-5e4ad8bea2a6-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:40:57.257515 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:57.257487    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc"
Jul 17 22:40:57.257728 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:57.257517    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc" event={"ID":"894a47b3-9073-4ddf-9998-5e4ad8bea2a6","Type":"ContainerDied","Data":"5fe6b3261c794ce6f1426831b3ca826acd7fd55370d889567d1eb0454433163f"}
Jul 17 22:40:57.257728 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:57.257584    2565 scope.go:117] "RemoveContainer" containerID="5ada5896cb80bc91206b74f637c204f9d6d5a9ca3bd2204a335a905ec0666d53"
Jul 17 22:40:57.276498 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:57.276465    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc"]
Jul 17 22:40:57.279362 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:57.279336    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-zfjlc"]
Jul 17 22:40:59.182588 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:40:59.182505    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="894a47b3-9073-4ddf-9998-5e4ad8bea2a6" path="/var/lib/kubelet/pods/894a47b3-9073-4ddf-9998-5e4ad8bea2a6/volumes"
Jul 17 22:41:05.492051 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492009    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q"]
Jul 17 22:41:05.492504 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492438    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab765028-f13d-4b75-8a4f-26b2ce6d2af5" containerName="revoke-keys"
Jul 17 22:41:05.492504 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492450    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab765028-f13d-4b75-8a4f-26b2ce6d2af5" containerName="revoke-keys"
Jul 17 22:41:05.492504 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492468    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="894a47b3-9073-4ddf-9998-5e4ad8bea2a6" containerName="revoke-keys"
Jul 17 22:41:05.492504 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492474    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="894a47b3-9073-4ddf-9998-5e4ad8bea2a6" containerName="revoke-keys"
Jul 17 22:41:05.492504 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492484    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="894a47b3-9073-4ddf-9998-5e4ad8bea2a6" containerName="revoke-keys"
Jul 17 22:41:05.492504 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492494    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="894a47b3-9073-4ddf-9998-5e4ad8bea2a6" containerName="revoke-keys"
Jul 17 22:41:05.492504 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492501    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95fb333d-9439-4f99-8116-87f7c3c20fe5" containerName="revoke-keys"
Jul 17 22:41:05.492504 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492506    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="95fb333d-9439-4f99-8116-87f7c3c20fe5" containerName="revoke-keys"
Jul 17 22:41:05.492818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492515    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95fb333d-9439-4f99-8116-87f7c3c20fe5" containerName="revoke-keys"
Jul 17 22:41:05.492818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492521    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="95fb333d-9439-4f99-8116-87f7c3c20fe5" containerName="revoke-keys"
Jul 17 22:41:05.492818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492527    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95fb333d-9439-4f99-8116-87f7c3c20fe5" containerName="revoke-keys"
Jul 17 22:41:05.492818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492532    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="95fb333d-9439-4f99-8116-87f7c3c20fe5" containerName="revoke-keys"
Jul 17 22:41:05.492818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492592    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="894a47b3-9073-4ddf-9998-5e4ad8bea2a6" containerName="revoke-keys"
Jul 17 22:41:05.492818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492600    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab765028-f13d-4b75-8a4f-26b2ce6d2af5" containerName="revoke-keys"
Jul 17 22:41:05.492818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492608    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="95fb333d-9439-4f99-8116-87f7c3c20fe5" containerName="revoke-keys"
Jul 17 22:41:05.492818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492613    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="894a47b3-9073-4ddf-9998-5e4ad8bea2a6" containerName="revoke-keys"
Jul 17 22:41:05.492818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.492619    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="95fb333d-9439-4f99-8116-87f7c3c20fe5" containerName="revoke-keys"
Jul 17 22:41:05.498017 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.498000    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q"
Jul 17 22:41:05.501025 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.501004    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:41:05.501956 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.501936    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:41:05.506301 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.506277    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q"]
Jul 17 22:41:05.522245 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.522209    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca65c04-5478-4e35-a5b4-99dee4765268-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q\" (UID: \"7ca65c04-5478-4e35-a5b4-99dee4765268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q"
Jul 17 22:41:05.622679 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.622647    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca65c04-5478-4e35-a5b4-99dee4765268-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q\" (UID: \"7ca65c04-5478-4e35-a5b4-99dee4765268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q"
Jul 17 22:41:05.623345 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.623323    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca65c04-5478-4e35-a5b4-99dee4765268-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q\" (UID: \"7ca65c04-5478-4e35-a5b4-99dee4765268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q"
Jul 17 22:41:05.810842 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.810808    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q"
Jul 17 22:41:05.980162 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:05.980132    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q"]
Jul 17 22:41:05.981957 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:41:05.981920    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca65c04_5478_4e35_a5b4_99dee4765268.slice/crio-3d180786529f6998c2717fb334d9efb52793273fa1876a675a723f792d4ceac6 WatchSource:0}: Error finding container 3d180786529f6998c2717fb334d9efb52793273fa1876a675a723f792d4ceac6: Status 404 returned error can't find the container with id 3d180786529f6998c2717fb334d9efb52793273fa1876a675a723f792d4ceac6
Jul 17 22:41:06.257672 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:06.257646    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:41:06.306408 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:06.306375    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q" event={"ID":"7ca65c04-5478-4e35-a5b4-99dee4765268","Type":"ContainerStarted","Data":"116b2f4ea453125d538f6014cd16874e6ccb7ce93c12f104a24c486ff8351514"}
Jul 17 22:41:06.306592 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:06.306437    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q" event={"ID":"7ca65c04-5478-4e35-a5b4-99dee4765268","Type":"ContainerStarted","Data":"3d180786529f6998c2717fb334d9efb52793273fa1876a675a723f792d4ceac6"}
Jul 17 22:41:06.323590 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:06.323536    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q" podStartSLOduration=1.32351983 podStartE2EDuration="1.32351983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:41:06.321396796 +0000 UTC m=+3329.710413475" watchObservedRunningTime="2026-07-17 22:41:06.32351983 +0000 UTC m=+3329.712536480"
Jul 17 22:41:07.312602 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:07.312567    2565 generic.go:356] "Generic (PLEG): container finished" podID="7ca65c04-5478-4e35-a5b4-99dee4765268" containerID="116b2f4ea453125d538f6014cd16874e6ccb7ce93c12f104a24c486ff8351514" exitCode=6
Jul 17 22:41:07.313106 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:07.312658    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q" event={"ID":"7ca65c04-5478-4e35-a5b4-99dee4765268","Type":"ContainerDied","Data":"116b2f4ea453125d538f6014cd16874e6ccb7ce93c12f104a24c486ff8351514"}
Jul 17 22:41:07.313106 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:07.312865    2565 scope.go:117] "RemoveContainer" containerID="116b2f4ea453125d538f6014cd16874e6ccb7ce93c12f104a24c486ff8351514"
Jul 17 22:41:08.319700 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:08.319667    2565 generic.go:356] "Generic (PLEG): container finished" podID="7ca65c04-5478-4e35-a5b4-99dee4765268" containerID="d0c3748d9ffc3a31654bd12e2f252485b63452027fa209dcfc07ec004bd58dad" exitCode=6
Jul 17 22:41:08.320132 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:08.319748    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q" event={"ID":"7ca65c04-5478-4e35-a5b4-99dee4765268","Type":"ContainerDied","Data":"d0c3748d9ffc3a31654bd12e2f252485b63452027fa209dcfc07ec004bd58dad"}
Jul 17 22:41:08.320132 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:08.319804    2565 scope.go:117] "RemoveContainer" containerID="116b2f4ea453125d538f6014cd16874e6ccb7ce93c12f104a24c486ff8351514"
Jul 17 22:41:08.320132 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:08.320040    2565 scope.go:117] "RemoveContainer" containerID="d0c3748d9ffc3a31654bd12e2f252485b63452027fa209dcfc07ec004bd58dad"
Jul 17 22:41:08.320343 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:41:08.320324    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q_odh-ai-gateway-infra(7ca65c04-5478-4e35-a5b4-99dee4765268)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q" podUID="7ca65c04-5478-4e35-a5b4-99dee4765268"
Jul 17 22:41:09.326748 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:09.326720    2565 scope.go:117] "RemoveContainer" containerID="d0c3748d9ffc3a31654bd12e2f252485b63452027fa209dcfc07ec004bd58dad"
Jul 17 22:41:09.327221 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:41:09.327004    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q_odh-ai-gateway-infra(7ca65c04-5478-4e35-a5b4-99dee4765268)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q" podUID="7ca65c04-5478-4e35-a5b4-99dee4765268"
Jul 17 22:41:10.746529 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:10.746489    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:41:17.035825 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:17.035784    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd"]
Jul 17 22:41:17.036288 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:17.036216    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="894a47b3-9073-4ddf-9998-5e4ad8bea2a6" containerName="revoke-keys"
Jul 17 22:41:17.036288 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:17.036229    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="894a47b3-9073-4ddf-9998-5e4ad8bea2a6" containerName="revoke-keys"
Jul 17 22:41:17.036288 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:17.036289    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="95fb333d-9439-4f99-8116-87f7c3c20fe5" containerName="revoke-keys"
Jul 17 22:41:17.036391 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:17.036299    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="894a47b3-9073-4ddf-9998-5e4ad8bea2a6" containerName="revoke-keys"
Jul 17 22:41:17.040898 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:17.040874    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd"
Jul 17 22:41:17.047965 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:17.047934    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd"]
Jul 17 22:41:17.142742 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:17.142706    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f558241-43ad-4317-80ea-43bf6882cf6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-msdhd\" (UID: \"3f558241-43ad-4317-80ea-43bf6882cf6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd"
Jul 17 22:41:17.244116 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:17.244042    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f558241-43ad-4317-80ea-43bf6882cf6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-msdhd\" (UID: \"3f558241-43ad-4317-80ea-43bf6882cf6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd"
Jul 17 22:41:17.244687 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:17.244668    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f558241-43ad-4317-80ea-43bf6882cf6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-msdhd\" (UID: \"3f558241-43ad-4317-80ea-43bf6882cf6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd"
Jul 17 22:41:17.353467 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:17.353382    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd"
Jul 17 22:41:17.501900 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:17.501875    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd"]
Jul 17 22:41:17.504939 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:41:17.504909    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f558241_43ad_4317_80ea_43bf6882cf6d.slice/crio-d664aa20b5750076ab82fdad9fd1730adf71c2634cbd64c7ce84734fa495717a WatchSource:0}: Error finding container d664aa20b5750076ab82fdad9fd1730adf71c2634cbd64c7ce84734fa495717a: Status 404 returned error can't find the container with id d664aa20b5750076ab82fdad9fd1730adf71c2634cbd64c7ce84734fa495717a
Jul 17 22:41:18.373574 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:18.373465    2565 generic.go:356] "Generic (PLEG): container finished" podID="3f558241-43ad-4317-80ea-43bf6882cf6d" containerID="cd93313d0c8fdc29d8ddef5b7f6aeda3ed9d950e561fc80c687f1ae8de142486" exitCode=6
Jul 17 22:41:18.373976 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:18.373559    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd" event={"ID":"3f558241-43ad-4317-80ea-43bf6882cf6d","Type":"ContainerDied","Data":"cd93313d0c8fdc29d8ddef5b7f6aeda3ed9d950e561fc80c687f1ae8de142486"}
Jul 17 22:41:18.373976 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:18.373627    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd" event={"ID":"3f558241-43ad-4317-80ea-43bf6882cf6d","Type":"ContainerStarted","Data":"d664aa20b5750076ab82fdad9fd1730adf71c2634cbd64c7ce84734fa495717a"}
Jul 17 22:41:18.373976 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:18.373708    2565 scope.go:117] "RemoveContainer" containerID="cd93313d0c8fdc29d8ddef5b7f6aeda3ed9d950e561fc80c687f1ae8de142486"
Jul 17 22:41:18.704676 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:18.704596    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:41:18.803169 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:18.803138    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:41:19.380186 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:19.380151    2565 generic.go:356] "Generic (PLEG): container finished" podID="3f558241-43ad-4317-80ea-43bf6882cf6d" containerID="53b17bd5f2f6e4046f913414fa4cad4482ec32497a14d31eaaa115ef7d37b4ee" exitCode=6
Jul 17 22:41:19.380622 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:19.380234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd" event={"ID":"3f558241-43ad-4317-80ea-43bf6882cf6d","Type":"ContainerDied","Data":"53b17bd5f2f6e4046f913414fa4cad4482ec32497a14d31eaaa115ef7d37b4ee"}
Jul 17 22:41:19.380622 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:19.380287    2565 scope.go:117] "RemoveContainer" containerID="cd93313d0c8fdc29d8ddef5b7f6aeda3ed9d950e561fc80c687f1ae8de142486"
Jul 17 22:41:19.380622 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:19.380485    2565 scope.go:117] "RemoveContainer" containerID="53b17bd5f2f6e4046f913414fa4cad4482ec32497a14d31eaaa115ef7d37b4ee"
Jul 17 22:41:19.380756 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:41:19.380729    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-msdhd_odh-ai-gateway-infra(3f558241-43ad-4317-80ea-43bf6882cf6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd" podUID="3f558241-43ad-4317-80ea-43bf6882cf6d"
Jul 17 22:41:20.387474 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:20.387446    2565 scope.go:117] "RemoveContainer" containerID="53b17bd5f2f6e4046f913414fa4cad4482ec32497a14d31eaaa115ef7d37b4ee"
Jul 17 22:41:20.387871 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:41:20.387665    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-msdhd_odh-ai-gateway-infra(3f558241-43ad-4317-80ea-43bf6882cf6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd" podUID="3f558241-43ad-4317-80ea-43bf6882cf6d"
Jul 17 22:41:21.177412 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:21.177381    2565 scope.go:117] "RemoveContainer" containerID="d0c3748d9ffc3a31654bd12e2f252485b63452027fa209dcfc07ec004bd58dad"
Jul 17 22:41:22.399131 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:22.399095    2565 generic.go:356] "Generic (PLEG): container finished" podID="7ca65c04-5478-4e35-a5b4-99dee4765268" containerID="7a1fb3c698012260b35c251873abbc23af982f4f42ad049bd6ea3f2dbe744c5f" exitCode=6
Jul 17 22:41:22.399568 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:22.399162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q" event={"ID":"7ca65c04-5478-4e35-a5b4-99dee4765268","Type":"ContainerDied","Data":"7a1fb3c698012260b35c251873abbc23af982f4f42ad049bd6ea3f2dbe744c5f"}
Jul 17 22:41:22.399568 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:22.399226    2565 scope.go:117] "RemoveContainer" containerID="d0c3748d9ffc3a31654bd12e2f252485b63452027fa209dcfc07ec004bd58dad"
Jul 17 22:41:22.399568 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:22.399533    2565 scope.go:117] "RemoveContainer" containerID="7a1fb3c698012260b35c251873abbc23af982f4f42ad049bd6ea3f2dbe744c5f"
Jul 17 22:41:22.399776 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:41:22.399751    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q_odh-ai-gateway-infra(7ca65c04-5478-4e35-a5b4-99dee4765268)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q" podUID="7ca65c04-5478-4e35-a5b4-99dee4765268"
Jul 17 22:41:23.396964 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:23.396925    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m"]
Jul 17 22:41:23.401628 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:23.401597    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m"
Jul 17 22:41:23.407983 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:23.407952    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m"]
Jul 17 22:41:23.423607 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:23.423577    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q"]
Jul 17 22:41:23.506985 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:23.506921    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de27024-5066-4da8-852d-67c37efad641-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-52r9m\" (UID: \"7de27024-5066-4da8-852d-67c37efad641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m"
Jul 17 22:41:23.563899 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:23.563867    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q"
Jul 17 22:41:23.608302 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:23.608267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de27024-5066-4da8-852d-67c37efad641-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-52r9m\" (UID: \"7de27024-5066-4da8-852d-67c37efad641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m"
Jul 17 22:41:23.608838 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:23.608820    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de27024-5066-4da8-852d-67c37efad641-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-52r9m\" (UID: \"7de27024-5066-4da8-852d-67c37efad641\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m"
Jul 17 22:41:23.708711 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:23.708624    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca65c04-5478-4e35-a5b4-99dee4765268-maas-api-service-ca\") pod \"7ca65c04-5478-4e35-a5b4-99dee4765268\" (UID: \"7ca65c04-5478-4e35-a5b4-99dee4765268\") "
Jul 17 22:41:23.709039 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:23.709010    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca65c04-5478-4e35-a5b4-99dee4765268-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ca65c04-5478-4e35-a5b4-99dee4765268" (UID: "7ca65c04-5478-4e35-a5b4-99dee4765268"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:41:23.714572 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:23.714548    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m"
Jul 17 22:41:23.809858 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:23.809827    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca65c04-5478-4e35-a5b4-99dee4765268-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:41:23.893951 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:23.893924    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m"]
Jul 17 22:41:23.896165 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:41:23.896132    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de27024_5066_4da8_852d_67c37efad641.slice/crio-503aebc37e7bb8532bc7639801e36133786e07a6d066939683e2aee31b0894de WatchSource:0}: Error finding container 503aebc37e7bb8532bc7639801e36133786e07a6d066939683e2aee31b0894de: Status 404 returned error can't find the container with id 503aebc37e7bb8532bc7639801e36133786e07a6d066939683e2aee31b0894de
Jul 17 22:41:24.414711 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:24.414611    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m" event={"ID":"7de27024-5066-4da8-852d-67c37efad641","Type":"ContainerStarted","Data":"9426a10b8aff901811226eae537e101b22f580dea010002280050beddcc1f83b"}
Jul 17 22:41:24.414711 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:24.414681    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m" event={"ID":"7de27024-5066-4da8-852d-67c37efad641","Type":"ContainerStarted","Data":"503aebc37e7bb8532bc7639801e36133786e07a6d066939683e2aee31b0894de"}
Jul 17 22:41:24.415997 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:24.415977    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q"
Jul 17 22:41:24.416139 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:24.416021    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q" event={"ID":"7ca65c04-5478-4e35-a5b4-99dee4765268","Type":"ContainerDied","Data":"3d180786529f6998c2717fb334d9efb52793273fa1876a675a723f792d4ceac6"}
Jul 17 22:41:24.416139 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:24.416099    2565 scope.go:117] "RemoveContainer" containerID="7a1fb3c698012260b35c251873abbc23af982f4f42ad049bd6ea3f2dbe744c5f"
Jul 17 22:41:24.434868 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:24.434810    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m" podStartSLOduration=1.434794885 podStartE2EDuration="1.434794885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:41:24.434608441 +0000 UTC m=+3347.823625093" watchObservedRunningTime="2026-07-17 22:41:24.434794885 +0000 UTC m=+3347.823811535"
Jul 17 22:41:24.451810 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:24.451766    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q"]
Jul 17 22:41:24.455618 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:24.455593    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-8fk2q"]
Jul 17 22:41:25.182203 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:25.182171    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ca65c04-5478-4e35-a5b4-99dee4765268" path="/var/lib/kubelet/pods/7ca65c04-5478-4e35-a5b4-99dee4765268/volumes"
Jul 17 22:41:25.423427 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:25.423390    2565 generic.go:356] "Generic (PLEG): container finished" podID="7de27024-5066-4da8-852d-67c37efad641" containerID="9426a10b8aff901811226eae537e101b22f580dea010002280050beddcc1f83b" exitCode=6
Jul 17 22:41:25.423899 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:25.423472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m" event={"ID":"7de27024-5066-4da8-852d-67c37efad641","Type":"ContainerDied","Data":"9426a10b8aff901811226eae537e101b22f580dea010002280050beddcc1f83b"}
Jul 17 22:41:25.423899 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:25.423703    2565 scope.go:117] "RemoveContainer" containerID="9426a10b8aff901811226eae537e101b22f580dea010002280050beddcc1f83b"
Jul 17 22:41:26.431625 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:26.431530    2565 generic.go:356] "Generic (PLEG): container finished" podID="7de27024-5066-4da8-852d-67c37efad641" containerID="719835ff9cdfec0b7a180212c22f676cde43ebdf47c9f6ad82ab7e8b7a479789" exitCode=6
Jul 17 22:41:26.431625 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:26.431591    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m" event={"ID":"7de27024-5066-4da8-852d-67c37efad641","Type":"ContainerDied","Data":"719835ff9cdfec0b7a180212c22f676cde43ebdf47c9f6ad82ab7e8b7a479789"}
Jul 17 22:41:26.432150 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:26.431642    2565 scope.go:117] "RemoveContainer" containerID="9426a10b8aff901811226eae537e101b22f580dea010002280050beddcc1f83b"
Jul 17 22:41:26.432150 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:26.431800    2565 scope.go:117] "RemoveContainer" containerID="719835ff9cdfec0b7a180212c22f676cde43ebdf47c9f6ad82ab7e8b7a479789"
Jul 17 22:41:26.432150 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:41:26.432040    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-52r9m_odh-ai-gateway-infra(7de27024-5066-4da8-852d-67c37efad641)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m" podUID="7de27024-5066-4da8-852d-67c37efad641"
Jul 17 22:41:27.439008 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:27.438979    2565 scope.go:117] "RemoveContainer" containerID="719835ff9cdfec0b7a180212c22f676cde43ebdf47c9f6ad82ab7e8b7a479789"
Jul 17 22:41:27.439418 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:41:27.439202    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-52r9m_odh-ai-gateway-infra(7de27024-5066-4da8-852d-67c37efad641)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m" podUID="7de27024-5066-4da8-852d-67c37efad641"
Jul 17 22:41:28.766670 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:28.766628    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:41:28.782028 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:28.781987    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp"]
Jul 17 22:41:28.782454 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:28.782436    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ca65c04-5478-4e35-a5b4-99dee4765268" containerName="revoke-keys"
Jul 17 22:41:28.782506 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:28.782456    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ca65c04-5478-4e35-a5b4-99dee4765268" containerName="revoke-keys"
Jul 17 22:41:28.782506 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:28.782481    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ca65c04-5478-4e35-a5b4-99dee4765268" containerName="revoke-keys"
Jul 17 22:41:28.782506 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:28.782487    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ca65c04-5478-4e35-a5b4-99dee4765268" containerName="revoke-keys"
Jul 17 22:41:28.782506 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:28.782497    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ca65c04-5478-4e35-a5b4-99dee4765268" containerName="revoke-keys"
Jul 17 22:41:28.782506 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:28.782502    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ca65c04-5478-4e35-a5b4-99dee4765268" containerName="revoke-keys"
Jul 17 22:41:28.782662 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:28.782578    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ca65c04-5478-4e35-a5b4-99dee4765268" containerName="revoke-keys"
Jul 17 22:41:28.782662 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:28.782586    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ca65c04-5478-4e35-a5b4-99dee4765268" containerName="revoke-keys"
Jul 17 22:41:28.782662 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:28.782595    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ca65c04-5478-4e35-a5b4-99dee4765268" containerName="revoke-keys"
Jul 17 22:41:28.787000 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:28.786973    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp"
Jul 17 22:41:28.797225 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:28.797187    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp"]
Jul 17 22:41:28.867627 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:28.867592    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:41:28.955559 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:28.955526    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8129bb-6f17-42c7-a4cb-cdeb424d772a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp\" (UID: \"eb8129bb-6f17-42c7-a4cb-cdeb424d772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp"
Jul 17 22:41:29.057047 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:29.056941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8129bb-6f17-42c7-a4cb-cdeb424d772a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp\" (UID: \"eb8129bb-6f17-42c7-a4cb-cdeb424d772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp"
Jul 17 22:41:29.057669 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:29.057642    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8129bb-6f17-42c7-a4cb-cdeb424d772a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp\" (UID: \"eb8129bb-6f17-42c7-a4cb-cdeb424d772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp"
Jul 17 22:41:29.097706 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:29.097667    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp"
Jul 17 22:41:29.242773 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:29.242744    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp"]
Jul 17 22:41:29.244744 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:41:29.244707    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8129bb_6f17_42c7_a4cb_cdeb424d772a.slice/crio-1bd8f60e10179889c37af868cdcca71f3618ddc399ba8156109e7d423728d154 WatchSource:0}: Error finding container 1bd8f60e10179889c37af868cdcca71f3618ddc399ba8156109e7d423728d154: Status 404 returned error can't find the container with id 1bd8f60e10179889c37af868cdcca71f3618ddc399ba8156109e7d423728d154
Jul 17 22:41:29.449582 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:29.449490    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp" event={"ID":"eb8129bb-6f17-42c7-a4cb-cdeb424d772a","Type":"ContainerStarted","Data":"1bd8f60e10179889c37af868cdcca71f3618ddc399ba8156109e7d423728d154"}
Jul 17 22:41:30.456079 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:30.456039    2565 generic.go:356] "Generic (PLEG): container finished" podID="eb8129bb-6f17-42c7-a4cb-cdeb424d772a" containerID="9e9fc5811cfea7ceecf6e345cdd4172dd19bb4844dcbd18e97e3ed16d5776b84" exitCode=6
Jul 17 22:41:30.456543 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:30.456142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp" event={"ID":"eb8129bb-6f17-42c7-a4cb-cdeb424d772a","Type":"ContainerDied","Data":"9e9fc5811cfea7ceecf6e345cdd4172dd19bb4844dcbd18e97e3ed16d5776b84"}
Jul 17 22:41:30.456543 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:30.456373    2565 scope.go:117] "RemoveContainer" containerID="9e9fc5811cfea7ceecf6e345cdd4172dd19bb4844dcbd18e97e3ed16d5776b84"
Jul 17 22:41:31.176626 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:31.176595    2565 scope.go:117] "RemoveContainer" containerID="53b17bd5f2f6e4046f913414fa4cad4482ec32497a14d31eaaa115ef7d37b4ee"
Jul 17 22:41:31.464290 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:31.464262    2565 generic.go:356] "Generic (PLEG): container finished" podID="eb8129bb-6f17-42c7-a4cb-cdeb424d772a" containerID="fd78addf0b1fe004321e1f4ae24a03b922746ed912f62210ee61b7870c55f01f" exitCode=6
Jul 17 22:41:31.464684 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:31.464325    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp" event={"ID":"eb8129bb-6f17-42c7-a4cb-cdeb424d772a","Type":"ContainerDied","Data":"fd78addf0b1fe004321e1f4ae24a03b922746ed912f62210ee61b7870c55f01f"}
Jul 17 22:41:31.464684 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:31.464376    2565 scope.go:117] "RemoveContainer" containerID="9e9fc5811cfea7ceecf6e345cdd4172dd19bb4844dcbd18e97e3ed16d5776b84"
Jul 17 22:41:31.464684 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:31.464586    2565 scope.go:117] "RemoveContainer" containerID="fd78addf0b1fe004321e1f4ae24a03b922746ed912f62210ee61b7870c55f01f"
Jul 17 22:41:31.464931 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:41:31.464906    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp_odh-ai-gateway-infra(eb8129bb-6f17-42c7-a4cb-cdeb424d772a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp" podUID="eb8129bb-6f17-42c7-a4cb-cdeb424d772a"
Jul 17 22:41:32.472187 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:32.472147    2565 generic.go:356] "Generic (PLEG): container finished" podID="3f558241-43ad-4317-80ea-43bf6882cf6d" containerID="c43acf5b3a72efb49c06e840a3135304c27a06afeaeef1fa362f782299fe04c9" exitCode=6
Jul 17 22:41:32.472627 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:32.472193    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd" event={"ID":"3f558241-43ad-4317-80ea-43bf6882cf6d","Type":"ContainerDied","Data":"c43acf5b3a72efb49c06e840a3135304c27a06afeaeef1fa362f782299fe04c9"}
Jul 17 22:41:32.472627 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:32.472298    2565 scope.go:117] "RemoveContainer" containerID="53b17bd5f2f6e4046f913414fa4cad4482ec32497a14d31eaaa115ef7d37b4ee"
Jul 17 22:41:32.472774 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:32.472696    2565 scope.go:117] "RemoveContainer" containerID="c43acf5b3a72efb49c06e840a3135304c27a06afeaeef1fa362f782299fe04c9"
Jul 17 22:41:32.472995 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:41:32.472966    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-msdhd_odh-ai-gateway-infra(3f558241-43ad-4317-80ea-43bf6882cf6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd" podUID="3f558241-43ad-4317-80ea-43bf6882cf6d"
Jul 17 22:41:32.474314 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:32.474297    2565 scope.go:117] "RemoveContainer" containerID="fd78addf0b1fe004321e1f4ae24a03b922746ed912f62210ee61b7870c55f01f"
Jul 17 22:41:32.474481 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:41:32.474461    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp_odh-ai-gateway-infra(eb8129bb-6f17-42c7-a4cb-cdeb424d772a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp" podUID="eb8129bb-6f17-42c7-a4cb-cdeb424d772a"
Jul 17 22:41:33.515732 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:33.515686    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd"]
Jul 17 22:41:33.658971 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:33.658939    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd"
Jul 17 22:41:33.802932 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:33.802833    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f558241-43ad-4317-80ea-43bf6882cf6d-maas-api-service-ca\") pod \"3f558241-43ad-4317-80ea-43bf6882cf6d\" (UID: \"3f558241-43ad-4317-80ea-43bf6882cf6d\") "
Jul 17 22:41:33.803259 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:33.803237    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f558241-43ad-4317-80ea-43bf6882cf6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f558241-43ad-4317-80ea-43bf6882cf6d" (UID: "3f558241-43ad-4317-80ea-43bf6882cf6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:41:33.904553 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:33.904520    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f558241-43ad-4317-80ea-43bf6882cf6d-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:41:34.486987 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:34.486952    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd" event={"ID":"3f558241-43ad-4317-80ea-43bf6882cf6d","Type":"ContainerDied","Data":"d664aa20b5750076ab82fdad9fd1730adf71c2634cbd64c7ce84734fa495717a"}
Jul 17 22:41:34.487212 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:34.486995    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd"
Jul 17 22:41:34.487212 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:34.487015    2565 scope.go:117] "RemoveContainer" containerID="c43acf5b3a72efb49c06e840a3135304c27a06afeaeef1fa362f782299fe04c9"
Jul 17 22:41:34.508836 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:34.508810    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd"]
Jul 17 22:41:34.515780 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:34.515756    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-msdhd"]
Jul 17 22:41:35.182716 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:35.182686    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f558241-43ad-4317-80ea-43bf6882cf6d" path="/var/lib/kubelet/pods/3f558241-43ad-4317-80ea-43bf6882cf6d/volumes"
Jul 17 22:41:35.252057 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:35.252023    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:41:40.177003 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:40.176965    2565 scope.go:117] "RemoveContainer" containerID="719835ff9cdfec0b7a180212c22f676cde43ebdf47c9f6ad82ab7e8b7a479789"
Jul 17 22:41:40.522819 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:40.522785    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m" event={"ID":"7de27024-5066-4da8-852d-67c37efad641","Type":"ContainerStarted","Data":"4836be51b978c2f2349ceea55d2351989f0a9a05f6c5158a123c884d29002372"}
Jul 17 22:41:41.529801 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:41.529767    2565 generic.go:356] "Generic (PLEG): container finished" podID="7de27024-5066-4da8-852d-67c37efad641" containerID="4836be51b978c2f2349ceea55d2351989f0a9a05f6c5158a123c884d29002372" exitCode=6
Jul 17 22:41:41.530260 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:41.529844    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m" event={"ID":"7de27024-5066-4da8-852d-67c37efad641","Type":"ContainerDied","Data":"4836be51b978c2f2349ceea55d2351989f0a9a05f6c5158a123c884d29002372"}
Jul 17 22:41:41.530260 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:41.529907    2565 scope.go:117] "RemoveContainer" containerID="719835ff9cdfec0b7a180212c22f676cde43ebdf47c9f6ad82ab7e8b7a479789"
Jul 17 22:41:41.530351 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:41.530316    2565 scope.go:117] "RemoveContainer" containerID="4836be51b978c2f2349ceea55d2351989f0a9a05f6c5158a123c884d29002372"
Jul 17 22:41:41.530547 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:41:41.530524    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-52r9m_odh-ai-gateway-infra(7de27024-5066-4da8-852d-67c37efad641)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m" podUID="7de27024-5066-4da8-852d-67c37efad641"
Jul 17 22:41:41.557241 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:41.557208    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m"]
Jul 17 22:41:42.678457 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:42.678432    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m"
Jul 17 22:41:42.688410 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:42.688386    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de27024-5066-4da8-852d-67c37efad641-maas-api-service-ca\") pod \"7de27024-5066-4da8-852d-67c37efad641\" (UID: \"7de27024-5066-4da8-852d-67c37efad641\") "
Jul 17 22:41:42.688738 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:42.688719    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de27024-5066-4da8-852d-67c37efad641-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7de27024-5066-4da8-852d-67c37efad641" (UID: "7de27024-5066-4da8-852d-67c37efad641"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:41:42.789168 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:42.789035    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7de27024-5066-4da8-852d-67c37efad641-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:41:43.001056 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:43.001021    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:41:43.062297 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:43.062203    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:41:43.543218 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:43.543187    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m"
Jul 17 22:41:43.543444 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:43.543230    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m" event={"ID":"7de27024-5066-4da8-852d-67c37efad641","Type":"ContainerDied","Data":"503aebc37e7bb8532bc7639801e36133786e07a6d066939683e2aee31b0894de"}
Jul 17 22:41:43.543444 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:43.543327    2565 scope.go:117] "RemoveContainer" containerID="4836be51b978c2f2349ceea55d2351989f0a9a05f6c5158a123c884d29002372"
Jul 17 22:41:43.562006 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:43.561974    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m"]
Jul 17 22:41:43.571953 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:43.571926    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-52r9m"]
Jul 17 22:41:45.182491 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:45.182457    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7de27024-5066-4da8-852d-67c37efad641" path="/var/lib/kubelet/pods/7de27024-5066-4da8-852d-67c37efad641/volumes"
Jul 17 22:41:47.179486 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:47.179461    2565 scope.go:117] "RemoveContainer" containerID="fd78addf0b1fe004321e1f4ae24a03b922746ed912f62210ee61b7870c55f01f"
Jul 17 22:41:47.565841 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:47.565800    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp" event={"ID":"eb8129bb-6f17-42c7-a4cb-cdeb424d772a","Type":"ContainerStarted","Data":"98a8e4199bd6018de02c1d6c9875b856ff0512d93ce8e5be3ed505f09d5d1e04"}
Jul 17 22:41:47.584639 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:47.584588    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp" podStartSLOduration=19.58456624 podStartE2EDuration="19.58456624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:41:47.581361359 +0000 UTC m=+3370.970378012" watchObservedRunningTime="2026-07-17 22:41:47.58456624 +0000 UTC m=+3370.973582892"
Jul 17 22:41:48.572680 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:48.572645    2565 generic.go:356] "Generic (PLEG): container finished" podID="eb8129bb-6f17-42c7-a4cb-cdeb424d772a" containerID="98a8e4199bd6018de02c1d6c9875b856ff0512d93ce8e5be3ed505f09d5d1e04" exitCode=6
Jul 17 22:41:48.573162 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:48.572716    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp" event={"ID":"eb8129bb-6f17-42c7-a4cb-cdeb424d772a","Type":"ContainerDied","Data":"98a8e4199bd6018de02c1d6c9875b856ff0512d93ce8e5be3ed505f09d5d1e04"}
Jul 17 22:41:48.573162 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:48.572776    2565 scope.go:117] "RemoveContainer" containerID="fd78addf0b1fe004321e1f4ae24a03b922746ed912f62210ee61b7870c55f01f"
Jul 17 22:41:48.573162 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:48.573059    2565 scope.go:117] "RemoveContainer" containerID="98a8e4199bd6018de02c1d6c9875b856ff0512d93ce8e5be3ed505f09d5d1e04"
Jul 17 22:41:48.573325 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:41:48.573303    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp_odh-ai-gateway-infra(eb8129bb-6f17-42c7-a4cb-cdeb424d772a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp" podUID="eb8129bb-6f17-42c7-a4cb-cdeb424d772a"
Jul 17 22:41:48.600809 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:48.600777    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp"]
Jul 17 22:41:49.724625 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:49.724598    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp"
Jul 17 22:41:49.750788 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:49.750759    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8129bb-6f17-42c7-a4cb-cdeb424d772a-maas-api-service-ca\") pod \"eb8129bb-6f17-42c7-a4cb-cdeb424d772a\" (UID: \"eb8129bb-6f17-42c7-a4cb-cdeb424d772a\") "
Jul 17 22:41:49.751151 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:49.751130    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8129bb-6f17-42c7-a4cb-cdeb424d772a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb8129bb-6f17-42c7-a4cb-cdeb424d772a" (UID: "eb8129bb-6f17-42c7-a4cb-cdeb424d772a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:41:49.852222 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:49.852122    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8129bb-6f17-42c7-a4cb-cdeb424d772a-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:41:50.139077 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:50.138986    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:41:50.585123 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:50.585095    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp"
Jul 17 22:41:50.585315 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:50.585133    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp" event={"ID":"eb8129bb-6f17-42c7-a4cb-cdeb424d772a","Type":"ContainerDied","Data":"1bd8f60e10179889c37af868cdcca71f3618ddc399ba8156109e7d423728d154"}
Jul 17 22:41:50.585315 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:50.585193    2565 scope.go:117] "RemoveContainer" containerID="98a8e4199bd6018de02c1d6c9875b856ff0512d93ce8e5be3ed505f09d5d1e04"
Jul 17 22:41:50.610605 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:50.610575    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp"]
Jul 17 22:41:50.614450 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:50.614419    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-29qhp"]
Jul 17 22:41:51.182577 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:51.182544    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb8129bb-6f17-42c7-a4cb-cdeb424d772a" path="/var/lib/kubelet/pods/eb8129bb-6f17-42c7-a4cb-cdeb424d772a/volumes"
Jul 17 22:41:55.535697 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.535663    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs"]
Jul 17 22:41:55.536090 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536042    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de27024-5066-4da8-852d-67c37efad641" containerName="revoke-keys"
Jul 17 22:41:55.536090 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536053    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de27024-5066-4da8-852d-67c37efad641" containerName="revoke-keys"
Jul 17 22:41:55.536090 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536078    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb8129bb-6f17-42c7-a4cb-cdeb424d772a" containerName="revoke-keys"
Jul 17 22:41:55.536090 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536086    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb8129bb-6f17-42c7-a4cb-cdeb424d772a" containerName="revoke-keys"
Jul 17 22:41:55.536238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536097    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f558241-43ad-4317-80ea-43bf6882cf6d" containerName="revoke-keys"
Jul 17 22:41:55.536238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536107    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f558241-43ad-4317-80ea-43bf6882cf6d" containerName="revoke-keys"
Jul 17 22:41:55.536238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536116    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f558241-43ad-4317-80ea-43bf6882cf6d" containerName="revoke-keys"
Jul 17 22:41:55.536238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536124    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f558241-43ad-4317-80ea-43bf6882cf6d" containerName="revoke-keys"
Jul 17 22:41:55.536238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536136    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb8129bb-6f17-42c7-a4cb-cdeb424d772a" containerName="revoke-keys"
Jul 17 22:41:55.536238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536142    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb8129bb-6f17-42c7-a4cb-cdeb424d772a" containerName="revoke-keys"
Jul 17 22:41:55.536238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536150    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb8129bb-6f17-42c7-a4cb-cdeb424d772a" containerName="revoke-keys"
Jul 17 22:41:55.536238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536155    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb8129bb-6f17-42c7-a4cb-cdeb424d772a" containerName="revoke-keys"
Jul 17 22:41:55.536238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536168    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f558241-43ad-4317-80ea-43bf6882cf6d" containerName="revoke-keys"
Jul 17 22:41:55.536238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536173    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f558241-43ad-4317-80ea-43bf6882cf6d" containerName="revoke-keys"
Jul 17 22:41:55.536238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536178    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de27024-5066-4da8-852d-67c37efad641" containerName="revoke-keys"
Jul 17 22:41:55.536238 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536189    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de27024-5066-4da8-852d-67c37efad641" containerName="revoke-keys"
Jul 17 22:41:55.536588 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536263    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de27024-5066-4da8-852d-67c37efad641" containerName="revoke-keys"
Jul 17 22:41:55.536588 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536271    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de27024-5066-4da8-852d-67c37efad641" containerName="revoke-keys"
Jul 17 22:41:55.536588 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536277    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb8129bb-6f17-42c7-a4cb-cdeb424d772a" containerName="revoke-keys"
Jul 17 22:41:55.536588 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536287    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f558241-43ad-4317-80ea-43bf6882cf6d" containerName="revoke-keys"
Jul 17 22:41:55.536588 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536293    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f558241-43ad-4317-80ea-43bf6882cf6d" containerName="revoke-keys"
Jul 17 22:41:55.536588 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536299    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb8129bb-6f17-42c7-a4cb-cdeb424d772a" containerName="revoke-keys"
Jul 17 22:41:55.536588 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.536305    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb8129bb-6f17-42c7-a4cb-cdeb424d772a" containerName="revoke-keys"
Jul 17 22:41:55.540711 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.540692    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs"
Jul 17 22:41:55.543595 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.543574    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:41:55.544685 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.544102    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:41:55.550427 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.550394    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs"]
Jul 17 22:41:55.612961 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.612930    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8aa01f7-6f7b-4136-bfa5-667cc928b036-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs\" (UID: \"e8aa01f7-6f7b-4136-bfa5-667cc928b036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs"
Jul 17 22:41:55.714477 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.714439    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8aa01f7-6f7b-4136-bfa5-667cc928b036-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs\" (UID: \"e8aa01f7-6f7b-4136-bfa5-667cc928b036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs"
Jul 17 22:41:55.715110 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.715055    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8aa01f7-6f7b-4136-bfa5-667cc928b036-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs\" (UID: \"e8aa01f7-6f7b-4136-bfa5-667cc928b036\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs"
Jul 17 22:41:55.856876 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.856791    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs"
Jul 17 22:41:55.998919 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:55.998859    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs"]
Jul 17 22:41:56.001369 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:41:56.001333    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8aa01f7_6f7b_4136_bfa5_667cc928b036.slice/crio-23a8828546c5df2ff363c31f1d4dccbcb553fc2eef1f73c40a73bf58b526c02f WatchSource:0}: Error finding container 23a8828546c5df2ff363c31f1d4dccbcb553fc2eef1f73c40a73bf58b526c02f: Status 404 returned error can't find the container with id 23a8828546c5df2ff363c31f1d4dccbcb553fc2eef1f73c40a73bf58b526c02f
Jul 17 22:41:56.620796 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:56.620758    2565 generic.go:356] "Generic (PLEG): container finished" podID="e8aa01f7-6f7b-4136-bfa5-667cc928b036" containerID="8ac385572f6af30871c1ee47591cc2efd41c256ed6d3a3879e43e1952c312348" exitCode=6
Jul 17 22:41:56.621295 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:56.620813    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs" event={"ID":"e8aa01f7-6f7b-4136-bfa5-667cc928b036","Type":"ContainerDied","Data":"8ac385572f6af30871c1ee47591cc2efd41c256ed6d3a3879e43e1952c312348"}
Jul 17 22:41:56.621295 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:56.620862    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs" event={"ID":"e8aa01f7-6f7b-4136-bfa5-667cc928b036","Type":"ContainerStarted","Data":"23a8828546c5df2ff363c31f1d4dccbcb553fc2eef1f73c40a73bf58b526c02f"}
Jul 17 22:41:56.621295 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:56.621119    2565 scope.go:117] "RemoveContainer" containerID="8ac385572f6af30871c1ee47591cc2efd41c256ed6d3a3879e43e1952c312348"
Jul 17 22:41:56.726701 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:56.726667    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:41:57.628214 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:57.628109    2565 generic.go:356] "Generic (PLEG): container finished" podID="e8aa01f7-6f7b-4136-bfa5-667cc928b036" containerID="9a3009ed1a449e6f8c3a72133a8dfe81eeefbab4d904b4edcd329fc0e0d5f447" exitCode=6
Jul 17 22:41:57.628685 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:57.628414    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs" event={"ID":"e8aa01f7-6f7b-4136-bfa5-667cc928b036","Type":"ContainerDied","Data":"9a3009ed1a449e6f8c3a72133a8dfe81eeefbab4d904b4edcd329fc0e0d5f447"}
Jul 17 22:41:57.628685 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:57.628511    2565 scope.go:117] "RemoveContainer" containerID="8ac385572f6af30871c1ee47591cc2efd41c256ed6d3a3879e43e1952c312348"
Jul 17 22:41:57.629209 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:57.629190    2565 scope.go:117] "RemoveContainer" containerID="9a3009ed1a449e6f8c3a72133a8dfe81eeefbab4d904b4edcd329fc0e0d5f447"
Jul 17 22:41:57.629640 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:41:57.629614    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs_odh-ai-gateway-infra(e8aa01f7-6f7b-4136-bfa5-667cc928b036)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs" podUID="e8aa01f7-6f7b-4136-bfa5-667cc928b036"
Jul 17 22:41:57.677659 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:57.677626    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:41:58.635632 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:41:58.635602    2565 scope.go:117] "RemoveContainer" containerID="9a3009ed1a449e6f8c3a72133a8dfe81eeefbab4d904b4edcd329fc0e0d5f447"
Jul 17 22:41:58.636008 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:41:58.635883    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs_odh-ai-gateway-infra(e8aa01f7-6f7b-4136-bfa5-667cc928b036)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs" podUID="e8aa01f7-6f7b-4136-bfa5-667cc928b036"
Jul 17 22:42:07.085498 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:07.085461    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll"]
Jul 17 22:42:07.085896 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:07.085882    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de27024-5066-4da8-852d-67c37efad641" containerName="revoke-keys"
Jul 17 22:42:07.085896 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:07.085896    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de27024-5066-4da8-852d-67c37efad641" containerName="revoke-keys"
Jul 17 22:42:07.085982 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:07.085973    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de27024-5066-4da8-852d-67c37efad641" containerName="revoke-keys"
Jul 17 22:42:07.086022 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:07.085984    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f558241-43ad-4317-80ea-43bf6882cf6d" containerName="revoke-keys"
Jul 17 22:42:07.090408 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:07.090387    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll"
Jul 17 22:42:07.097277 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:07.097245    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll"]
Jul 17 22:42:07.219678 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:07.219625    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f182b2a5-ed43-48a6-95ce-7d6200238029-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-x95ll\" (UID: \"f182b2a5-ed43-48a6-95ce-7d6200238029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll"
Jul 17 22:42:07.320164 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:07.320125    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f182b2a5-ed43-48a6-95ce-7d6200238029-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-x95ll\" (UID: \"f182b2a5-ed43-48a6-95ce-7d6200238029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll"
Jul 17 22:42:07.320755 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:07.320736    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f182b2a5-ed43-48a6-95ce-7d6200238029-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-x95ll\" (UID: \"f182b2a5-ed43-48a6-95ce-7d6200238029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll"
Jul 17 22:42:07.404572 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:07.404472    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll"
Jul 17 22:42:07.545345 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:07.545316    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll"]
Jul 17 22:42:07.547230 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:42:07.547196    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf182b2a5_ed43_48a6_95ce_7d6200238029.slice/crio-78041ee0610f91028e560b46990281a7fcec02e89312f0b588392e195eff3035 WatchSource:0}: Error finding container 78041ee0610f91028e560b46990281a7fcec02e89312f0b588392e195eff3035: Status 404 returned error can't find the container with id 78041ee0610f91028e560b46990281a7fcec02e89312f0b588392e195eff3035
Jul 17 22:42:07.681513 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:07.681423    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll" event={"ID":"f182b2a5-ed43-48a6-95ce-7d6200238029","Type":"ContainerStarted","Data":"78041ee0610f91028e560b46990281a7fcec02e89312f0b588392e195eff3035"}
Jul 17 22:42:08.688150 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:08.688052    2565 generic.go:356] "Generic (PLEG): container finished" podID="f182b2a5-ed43-48a6-95ce-7d6200238029" containerID="27b5b8d299a2f8626d0cd7a4d7bb386d0cd7a3c4681cbe111c66d26f31f6b5ce" exitCode=6
Jul 17 22:42:08.688150 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:08.688106    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll" event={"ID":"f182b2a5-ed43-48a6-95ce-7d6200238029","Type":"ContainerDied","Data":"27b5b8d299a2f8626d0cd7a4d7bb386d0cd7a3c4681cbe111c66d26f31f6b5ce"}
Jul 17 22:42:08.688539 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:08.688325    2565 scope.go:117] "RemoveContainer" containerID="27b5b8d299a2f8626d0cd7a4d7bb386d0cd7a3c4681cbe111c66d26f31f6b5ce"
Jul 17 22:42:09.254232 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:09.254195    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:42:09.694884 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:09.694793    2565 generic.go:356] "Generic (PLEG): container finished" podID="f182b2a5-ed43-48a6-95ce-7d6200238029" containerID="b54898b8cdd93c2b50c96ca42324fe81667f69c7378c48cf92a3608d7d7a1632" exitCode=6
Jul 17 22:42:09.695340 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:09.694881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll" event={"ID":"f182b2a5-ed43-48a6-95ce-7d6200238029","Type":"ContainerDied","Data":"b54898b8cdd93c2b50c96ca42324fe81667f69c7378c48cf92a3608d7d7a1632"}
Jul 17 22:42:09.695340 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:09.694923    2565 scope.go:117] "RemoveContainer" containerID="27b5b8d299a2f8626d0cd7a4d7bb386d0cd7a3c4681cbe111c66d26f31f6b5ce"
Jul 17 22:42:09.695340 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:09.695044    2565 scope.go:117] "RemoveContainer" containerID="b54898b8cdd93c2b50c96ca42324fe81667f69c7378c48cf92a3608d7d7a1632"
Jul 17 22:42:09.695340 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:42:09.695305    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-x95ll_odh-ai-gateway-infra(f182b2a5-ed43-48a6-95ce-7d6200238029)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll" podUID="f182b2a5-ed43-48a6-95ce-7d6200238029"
Jul 17 22:42:10.701908 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:10.701881    2565 scope.go:117] "RemoveContainer" containerID="b54898b8cdd93c2b50c96ca42324fe81667f69c7378c48cf92a3608d7d7a1632"
Jul 17 22:42:10.702327 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:42:10.702139    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-x95ll_odh-ai-gateway-infra(f182b2a5-ed43-48a6-95ce-7d6200238029)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll" podUID="f182b2a5-ed43-48a6-95ce-7d6200238029"
Jul 17 22:42:13.177183 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:13.177150    2565 scope.go:117] "RemoveContainer" containerID="9a3009ed1a449e6f8c3a72133a8dfe81eeefbab4d904b4edcd329fc0e0d5f447"
Jul 17 22:42:13.719191 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:13.719105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs" event={"ID":"e8aa01f7-6f7b-4136-bfa5-667cc928b036","Type":"ContainerStarted","Data":"68652667c23088f4222e0d8d052e6b2dca00f9557a0fffe8a44bdbd8abaf180a"}
Jul 17 22:42:13.742038 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:13.741979    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs" podStartSLOduration=18.741962076 podStartE2EDuration="18.741962076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:42:13.738706226 +0000 UTC m=+3397.127722878" watchObservedRunningTime="2026-07-17 22:42:13.741962076 +0000 UTC m=+3397.130978726"
Jul 17 22:42:14.725890 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:14.725851    2565 generic.go:356] "Generic (PLEG): container finished" podID="e8aa01f7-6f7b-4136-bfa5-667cc928b036" containerID="68652667c23088f4222e0d8d052e6b2dca00f9557a0fffe8a44bdbd8abaf180a" exitCode=6
Jul 17 22:42:14.726431 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:14.725919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs" event={"ID":"e8aa01f7-6f7b-4136-bfa5-667cc928b036","Type":"ContainerDied","Data":"68652667c23088f4222e0d8d052e6b2dca00f9557a0fffe8a44bdbd8abaf180a"}
Jul 17 22:42:14.726431 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:14.725982    2565 scope.go:117] "RemoveContainer" containerID="9a3009ed1a449e6f8c3a72133a8dfe81eeefbab4d904b4edcd329fc0e0d5f447"
Jul 17 22:42:14.726431 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:14.726341    2565 scope.go:117] "RemoveContainer" containerID="68652667c23088f4222e0d8d052e6b2dca00f9557a0fffe8a44bdbd8abaf180a"
Jul 17 22:42:14.726639 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:42:14.726619    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs_odh-ai-gateway-infra(e8aa01f7-6f7b-4136-bfa5-667cc928b036)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs" podUID="e8aa01f7-6f7b-4136-bfa5-667cc928b036"
Jul 17 22:42:14.749696 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:14.749664    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs"]
Jul 17 22:42:15.875765 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:15.875739    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs"
Jul 17 22:42:16.006220 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:16.006185    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8aa01f7-6f7b-4136-bfa5-667cc928b036-maas-api-service-ca\") pod \"e8aa01f7-6f7b-4136-bfa5-667cc928b036\" (UID: \"e8aa01f7-6f7b-4136-bfa5-667cc928b036\") "
Jul 17 22:42:16.006552 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:16.006531    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8aa01f7-6f7b-4136-bfa5-667cc928b036-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e8aa01f7-6f7b-4136-bfa5-667cc928b036" (UID: "e8aa01f7-6f7b-4136-bfa5-667cc928b036"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:42:16.107296 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:16.107255    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8aa01f7-6f7b-4136-bfa5-667cc928b036-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:42:16.739180 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:16.739145    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs"
Jul 17 22:42:16.739180 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:16.739176    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs" event={"ID":"e8aa01f7-6f7b-4136-bfa5-667cc928b036","Type":"ContainerDied","Data":"23a8828546c5df2ff363c31f1d4dccbcb553fc2eef1f73c40a73bf58b526c02f"}
Jul 17 22:42:16.739418 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:16.739240    2565 scope.go:117] "RemoveContainer" containerID="68652667c23088f4222e0d8d052e6b2dca00f9557a0fffe8a44bdbd8abaf180a"
Jul 17 22:42:16.764974 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:16.764929    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs"]
Jul 17 22:42:16.770274 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:16.770248    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-w6hqs"]
Jul 17 22:42:17.185018 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:17.184983    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8aa01f7-6f7b-4136-bfa5-667cc928b036" path="/var/lib/kubelet/pods/e8aa01f7-6f7b-4136-bfa5-667cc928b036/volumes"
Jul 17 22:42:17.597308 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:17.597261    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:42:17.693620 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:17.693569    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:42:23.177096 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:23.177044    2565 scope.go:117] "RemoveContainer" containerID="b54898b8cdd93c2b50c96ca42324fe81667f69c7378c48cf92a3608d7d7a1632"
Jul 17 22:42:23.777996 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:23.777968    2565 generic.go:356] "Generic (PLEG): container finished" podID="f182b2a5-ed43-48a6-95ce-7d6200238029" containerID="6c5d050db82733bb573cc5a213c26442e16a57bbeb782efe562d593e461f0fb9" exitCode=6
Jul 17 22:42:23.778136 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:23.778038    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll" event={"ID":"f182b2a5-ed43-48a6-95ce-7d6200238029","Type":"ContainerDied","Data":"6c5d050db82733bb573cc5a213c26442e16a57bbeb782efe562d593e461f0fb9"}
Jul 17 22:42:23.778316 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:23.778302    2565 scope.go:117] "RemoveContainer" containerID="b54898b8cdd93c2b50c96ca42324fe81667f69c7378c48cf92a3608d7d7a1632"
Jul 17 22:42:23.778615 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:23.778599    2565 scope.go:117] "RemoveContainer" containerID="6c5d050db82733bb573cc5a213c26442e16a57bbeb782efe562d593e461f0fb9"
Jul 17 22:42:23.778836 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:42:23.778818    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-x95ll_odh-ai-gateway-infra(f182b2a5-ed43-48a6-95ce-7d6200238029)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll" podUID="f182b2a5-ed43-48a6-95ce-7d6200238029"
Jul 17 22:42:24.803393 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:24.803356    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll"]
Jul 17 22:42:24.945529 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:24.945502    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll"
Jul 17 22:42:25.089367 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:25.089259    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f182b2a5-ed43-48a6-95ce-7d6200238029-maas-api-service-ca\") pod \"f182b2a5-ed43-48a6-95ce-7d6200238029\" (UID: \"f182b2a5-ed43-48a6-95ce-7d6200238029\") "
Jul 17 22:42:25.089631 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:25.089607    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f182b2a5-ed43-48a6-95ce-7d6200238029-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f182b2a5-ed43-48a6-95ce-7d6200238029" (UID: "f182b2a5-ed43-48a6-95ce-7d6200238029"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:42:25.190223 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:25.190198    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f182b2a5-ed43-48a6-95ce-7d6200238029-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:42:25.791109 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:25.791043    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll" event={"ID":"f182b2a5-ed43-48a6-95ce-7d6200238029","Type":"ContainerDied","Data":"78041ee0610f91028e560b46990281a7fcec02e89312f0b588392e195eff3035"}
Jul 17 22:42:25.791319 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:25.791135    2565 scope.go:117] "RemoveContainer" containerID="6c5d050db82733bb573cc5a213c26442e16a57bbeb782efe562d593e461f0fb9"
Jul 17 22:42:25.791319 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:25.791092    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll"
Jul 17 22:42:25.807752 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:25.807725    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll"]
Jul 17 22:42:25.814023 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:25.813998    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-x95ll"]
Jul 17 22:42:27.182682 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:27.182643    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f182b2a5-ed43-48a6-95ce-7d6200238029" path="/var/lib/kubelet/pods/f182b2a5-ed43-48a6-95ce-7d6200238029/volumes"
Jul 17 22:42:27.360372 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:27.360338    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:42:31.151003 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:31.150924    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:42:31.924832 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:31.924797    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:42:38.079390 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:38.079349    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:42:38.180245 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:38.180161    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:42:42.606214 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:42.606171    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"]
Jul 17 22:42:42.606704 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:42.606378    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x" podUID="722b2832-2199-480b-ac12-f84cc1beb2dc" containerName="istio-proxy" containerID="cri-o://9f9babfa55438f6a05654daedc9fc083db0c7334854761862ab40f223c4511e4" gracePeriod=30
Jul 17 22:42:42.624726 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:42.624698    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:42:42.846124 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:42.846093    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:42:43.602702 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:43.602657    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:42:43.732747 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:43.732711    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:42:44.694058 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:44.694020    2565 patch_prober.go:28] interesting pod/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 22:42:44.694332 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:44.694128    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x" podUID="722b2832-2199-480b-ac12-f84cc1beb2dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 22:42:47.868984 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:47.868916    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:42:47.909683 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:47.909644    2565 generic.go:356] "Generic (PLEG): container finished" podID="722b2832-2199-480b-ac12-f84cc1beb2dc" containerID="9f9babfa55438f6a05654daedc9fc083db0c7334854761862ab40f223c4511e4" exitCode=0
Jul 17 22:42:47.909891 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:47.909693    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"
Jul 17 22:42:47.909891 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:47.909731    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x" event={"ID":"722b2832-2199-480b-ac12-f84cc1beb2dc","Type":"ContainerDied","Data":"9f9babfa55438f6a05654daedc9fc083db0c7334854761862ab40f223c4511e4"}
Jul 17 22:42:47.909891 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:47.909798    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x" event={"ID":"722b2832-2199-480b-ac12-f84cc1beb2dc","Type":"ContainerDied","Data":"bed0c56aba1aaa2bfac0693309201e7da58c12803fd62c75a110c1b7c01f2976"}
Jul 17 22:42:47.909891 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:47.909831    2565 scope.go:117] "RemoveContainer" containerID="9f9babfa55438f6a05654daedc9fc083db0c7334854761862ab40f223c4511e4"
Jul 17 22:42:47.918888 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:47.918872    2565 scope.go:117] "RemoveContainer" containerID="3fd9fe91b8f0ec045d01a0712bba2f2293edc91e278710af560e1dfb72befdc0"
Jul 17 22:42:47.927107 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:47.927087    2565 scope.go:117] "RemoveContainer" containerID="9f9babfa55438f6a05654daedc9fc083db0c7334854761862ab40f223c4511e4"
Jul 17 22:42:47.927365 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:42:47.927346    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9babfa55438f6a05654daedc9fc083db0c7334854761862ab40f223c4511e4\": container with ID starting with 9f9babfa55438f6a05654daedc9fc083db0c7334854761862ab40f223c4511e4 not found: ID does not exist" containerID="9f9babfa55438f6a05654daedc9fc083db0c7334854761862ab40f223c4511e4"
Jul 17 22:42:47.927412 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:47.927374    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9babfa55438f6a05654daedc9fc083db0c7334854761862ab40f223c4511e4"} err="failed to get container status \"9f9babfa55438f6a05654daedc9fc083db0c7334854761862ab40f223c4511e4\": rpc error: code = NotFound desc = could not find container \"9f9babfa55438f6a05654daedc9fc083db0c7334854761862ab40f223c4511e4\": container with ID starting with 9f9babfa55438f6a05654daedc9fc083db0c7334854761862ab40f223c4511e4 not found: ID does not exist"
Jul 17 22:42:47.927412 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:47.927393    2565 scope.go:117] "RemoveContainer" containerID="3fd9fe91b8f0ec045d01a0712bba2f2293edc91e278710af560e1dfb72befdc0"
Jul 17 22:42:47.927619 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:42:47.927598    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd9fe91b8f0ec045d01a0712bba2f2293edc91e278710af560e1dfb72befdc0\": container with ID starting with 3fd9fe91b8f0ec045d01a0712bba2f2293edc91e278710af560e1dfb72befdc0 not found: ID does not exist" containerID="3fd9fe91b8f0ec045d01a0712bba2f2293edc91e278710af560e1dfb72befdc0"
Jul 17 22:42:47.927676 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:47.927625    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd9fe91b8f0ec045d01a0712bba2f2293edc91e278710af560e1dfb72befdc0"} err="failed to get container status \"3fd9fe91b8f0ec045d01a0712bba2f2293edc91e278710af560e1dfb72befdc0\": rpc error: code = NotFound desc = could not find container \"3fd9fe91b8f0ec045d01a0712bba2f2293edc91e278710af560e1dfb72befdc0\": container with ID starting with 3fd9fe91b8f0ec045d01a0712bba2f2293edc91e278710af560e1dfb72befdc0 not found: ID does not exist"
Jul 17 22:42:48.008538 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.008502    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722b2832-2199-480b-ac12-f84cc1beb2dc-istiod-ca-cert\") pod \"722b2832-2199-480b-ac12-f84cc1beb2dc\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") "
Jul 17 22:42:48.008538 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.008551    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-credential-socket\") pod \"722b2832-2199-480b-ac12-f84cc1beb2dc\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") "
Jul 17 22:42:48.008750 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.008576    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-workload-socket\") pod \"722b2832-2199-480b-ac12-f84cc1beb2dc\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") "
Jul 17 22:42:48.008750 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.008601    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsq52\" (UniqueName: \"kubernetes.io/projected/722b2832-2199-480b-ac12-f84cc1beb2dc-kube-api-access-wsq52\") pod \"722b2832-2199-480b-ac12-f84cc1beb2dc\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") "
Jul 17 22:42:48.008750 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.008714    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-podinfo\") pod \"722b2832-2199-480b-ac12-f84cc1beb2dc\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") "
Jul 17 22:42:48.008886 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.008771    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-workload-certs\") pod \"722b2832-2199-480b-ac12-f84cc1beb2dc\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") "
Jul 17 22:42:48.008886 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.008800    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-token\") pod \"722b2832-2199-480b-ac12-f84cc1beb2dc\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") "
Jul 17 22:42:48.008886 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.008829    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "722b2832-2199-480b-ac12-f84cc1beb2dc" (UID: "722b2832-2199-480b-ac12-f84cc1beb2dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:42:48.009048 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.008903    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-data\") pod \"722b2832-2199-480b-ac12-f84cc1beb2dc\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") "
Jul 17 22:42:48.009048 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.008933    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-envoy\") pod \"722b2832-2199-480b-ac12-f84cc1beb2dc\" (UID: \"722b2832-2199-480b-ac12-f84cc1beb2dc\") "
Jul 17 22:42:48.009048 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.008948    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "722b2832-2199-480b-ac12-f84cc1beb2dc" (UID: "722b2832-2199-480b-ac12-f84cc1beb2dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:42:48.009930 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.008979    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/722b2832-2199-480b-ac12-f84cc1beb2dc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "722b2832-2199-480b-ac12-f84cc1beb2dc" (UID: "722b2832-2199-480b-ac12-f84cc1beb2dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:42:48.009930 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.009055    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "722b2832-2199-480b-ac12-f84cc1beb2dc" (UID: "722b2832-2199-480b-ac12-f84cc1beb2dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:42:48.009930 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.009330    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-workload-certs\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:42:48.009930 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.009360    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722b2832-2199-480b-ac12-f84cc1beb2dc-istiod-ca-cert\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:42:48.009930 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.009375    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-credential-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:42:48.009930 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.009388    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-workload-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:42:48.010306 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.009956    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-data" (OuterVolumeSpecName: "istio-data") pod "722b2832-2199-480b-ac12-f84cc1beb2dc" (UID: "722b2832-2199-480b-ac12-f84cc1beb2dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:42:48.011008 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.010982    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "722b2832-2199-480b-ac12-f84cc1beb2dc" (UID: "722b2832-2199-480b-ac12-f84cc1beb2dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:42:48.011008 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.011000    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722b2832-2199-480b-ac12-f84cc1beb2dc-kube-api-access-wsq52" (OuterVolumeSpecName: "kube-api-access-wsq52") pod "722b2832-2199-480b-ac12-f84cc1beb2dc" (UID: "722b2832-2199-480b-ac12-f84cc1beb2dc"). InnerVolumeSpecName "kube-api-access-wsq52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:42:48.011263 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.011154    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "722b2832-2199-480b-ac12-f84cc1beb2dc" (UID: "722b2832-2199-480b-ac12-f84cc1beb2dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:42:48.011263 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.011210    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-token" (OuterVolumeSpecName: "istio-token") pod "722b2832-2199-480b-ac12-f84cc1beb2dc" (UID: "722b2832-2199-480b-ac12-f84cc1beb2dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:42:48.109864 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.109826    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsq52\" (UniqueName: \"kubernetes.io/projected/722b2832-2199-480b-ac12-f84cc1beb2dc-kube-api-access-wsq52\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:42:48.109864 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.109857    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-podinfo\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:42:48.109864 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.109867    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-token\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:42:48.109864 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.109874    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-data\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:42:48.110173 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.109884    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722b2832-2199-480b-ac12-f84cc1beb2dc-istio-envoy\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:42:48.232734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.232701    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"]
Jul 17 22:42:48.236929 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:48.236903    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-a56653-openshift-default-6786985bff-njl8x"]
Jul 17 22:42:49.183105 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:49.183049    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="722b2832-2199-480b-ac12-f84cc1beb2dc" path="/var/lib/kubelet/pods/722b2832-2199-480b-ac12-f84cc1beb2dc/volumes"
Jul 17 22:42:53.453172 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:53.453128    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:42:55.583131 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583089    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs"]
Jul 17 22:42:55.583523 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583507    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8aa01f7-6f7b-4136-bfa5-667cc928b036" containerName="revoke-keys"
Jul 17 22:42:55.583523 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583520    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8aa01f7-6f7b-4136-bfa5-667cc928b036" containerName="revoke-keys"
Jul 17 22:42:55.583601 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583533    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="722b2832-2199-480b-ac12-f84cc1beb2dc" containerName="istio-proxy"
Jul 17 22:42:55.583601 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583539    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="722b2832-2199-480b-ac12-f84cc1beb2dc" containerName="istio-proxy"
Jul 17 22:42:55.583601 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583553    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8aa01f7-6f7b-4136-bfa5-667cc928b036" containerName="revoke-keys"
Jul 17 22:42:55.583601 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583559    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8aa01f7-6f7b-4136-bfa5-667cc928b036" containerName="revoke-keys"
Jul 17 22:42:55.583601 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583586    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f182b2a5-ed43-48a6-95ce-7d6200238029" containerName="revoke-keys"
Jul 17 22:42:55.583601 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583593    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f182b2a5-ed43-48a6-95ce-7d6200238029" containerName="revoke-keys"
Jul 17 22:42:55.583601 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583600    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f182b2a5-ed43-48a6-95ce-7d6200238029" containerName="revoke-keys"
Jul 17 22:42:55.583818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583606    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f182b2a5-ed43-48a6-95ce-7d6200238029" containerName="revoke-keys"
Jul 17 22:42:55.583818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583616    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="722b2832-2199-480b-ac12-f84cc1beb2dc" containerName="istio-proxy"
Jul 17 22:42:55.583818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583621    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="722b2832-2199-480b-ac12-f84cc1beb2dc" containerName="istio-proxy"
Jul 17 22:42:55.583818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583627    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f182b2a5-ed43-48a6-95ce-7d6200238029" containerName="revoke-keys"
Jul 17 22:42:55.583818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583632    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f182b2a5-ed43-48a6-95ce-7d6200238029" containerName="revoke-keys"
Jul 17 22:42:55.583818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583694    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8aa01f7-6f7b-4136-bfa5-667cc928b036" containerName="revoke-keys"
Jul 17 22:42:55.583818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583702    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="722b2832-2199-480b-ac12-f84cc1beb2dc" containerName="istio-proxy"
Jul 17 22:42:55.583818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583708    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8aa01f7-6f7b-4136-bfa5-667cc928b036" containerName="revoke-keys"
Jul 17 22:42:55.583818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583716    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f182b2a5-ed43-48a6-95ce-7d6200238029" containerName="revoke-keys"
Jul 17 22:42:55.583818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583722    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="722b2832-2199-480b-ac12-f84cc1beb2dc" containerName="istio-proxy"
Jul 17 22:42:55.583818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583728    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8aa01f7-6f7b-4136-bfa5-667cc928b036" containerName="revoke-keys"
Jul 17 22:42:55.583818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.583742    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f182b2a5-ed43-48a6-95ce-7d6200238029" containerName="revoke-keys"
Jul 17 22:42:55.586512 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.586495    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs"
Jul 17 22:42:55.589033 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.589012    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:42:55.589033 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.589024    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:42:55.594526 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.594483    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs"]
Jul 17 22:42:55.683261 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.683219    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c074c4d3-ba15-4058-a0b4-812e4e543e10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs\" (UID: \"c074c4d3-ba15-4058-a0b4-812e4e543e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs"
Jul 17 22:42:55.784533 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.784494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c074c4d3-ba15-4058-a0b4-812e4e543e10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs\" (UID: \"c074c4d3-ba15-4058-a0b4-812e4e543e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs"
Jul 17 22:42:55.785145 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.785126    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c074c4d3-ba15-4058-a0b4-812e4e543e10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs\" (UID: \"c074c4d3-ba15-4058-a0b4-812e4e543e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs"
Jul 17 22:42:55.898521 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:55.898426    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs"
Jul 17 22:42:56.047853 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:56.047831    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs"]
Jul 17 22:42:56.050420 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:42:56.050391    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc074c4d3_ba15_4058_a0b4_812e4e543e10.slice/crio-edfd26e142c950f34d436830776d86b099f20fdf9df867058e5889fae26284bb WatchSource:0}: Error finding container edfd26e142c950f34d436830776d86b099f20fdf9df867058e5889fae26284bb: Status 404 returned error can't find the container with id edfd26e142c950f34d436830776d86b099f20fdf9df867058e5889fae26284bb
Jul 17 22:42:56.844882 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:56.844846    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2g44p"]
Jul 17 22:42:56.956303 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:56.956209    2565 generic.go:356] "Generic (PLEG): container finished" podID="c074c4d3-ba15-4058-a0b4-812e4e543e10" containerID="ec70cd3efb6b4798f672384fb042f172d9f67e88d1a2ecc6a1cae9bccd45d873" exitCode=6
Jul 17 22:42:56.956303 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:56.956269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs" event={"ID":"c074c4d3-ba15-4058-a0b4-812e4e543e10","Type":"ContainerDied","Data":"ec70cd3efb6b4798f672384fb042f172d9f67e88d1a2ecc6a1cae9bccd45d873"}
Jul 17 22:42:56.956519 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:56.956335    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs" event={"ID":"c074c4d3-ba15-4058-a0b4-812e4e543e10","Type":"ContainerStarted","Data":"edfd26e142c950f34d436830776d86b099f20fdf9df867058e5889fae26284bb"}
Jul 17 22:42:56.956519 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:56.956472    2565 scope.go:117] "RemoveContainer" containerID="ec70cd3efb6b4798f672384fb042f172d9f67e88d1a2ecc6a1cae9bccd45d873"
Jul 17 22:42:57.131131 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.131092    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q"]
Jul 17 22:42:57.131479 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.131466    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8aa01f7-6f7b-4136-bfa5-667cc928b036" containerName="revoke-keys"
Jul 17 22:42:57.131546 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.131481    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8aa01f7-6f7b-4136-bfa5-667cc928b036" containerName="revoke-keys"
Jul 17 22:42:57.131592 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.131560    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f182b2a5-ed43-48a6-95ce-7d6200238029" containerName="revoke-keys"
Jul 17 22:42:57.135834 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.135816    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q"
Jul 17 22:42:57.141734 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.141705    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q"]
Jul 17 22:42:57.300544 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.300508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6c2901-7a95-49d9-aba6-5315349636ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q\" (UID: \"ed6c2901-7a95-49d9-aba6-5315349636ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q"
Jul 17 22:42:57.401921 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.401886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6c2901-7a95-49d9-aba6-5315349636ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q\" (UID: \"ed6c2901-7a95-49d9-aba6-5315349636ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q"
Jul 17 22:42:57.402551 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.402527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6c2901-7a95-49d9-aba6-5315349636ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q\" (UID: \"ed6c2901-7a95-49d9-aba6-5315349636ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q"
Jul 17 22:42:57.447698 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.447656    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q"
Jul 17 22:42:57.600167 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.600135    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q"]
Jul 17 22:42:57.603022 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:42:57.602990    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6c2901_7a95_49d9_aba6_5315349636ab.slice/crio-04e2892022ae4054c8c7bc861ac2de58728d493f209afaf3d5e43ea2117230c9 WatchSource:0}: Error finding container 04e2892022ae4054c8c7bc861ac2de58728d493f209afaf3d5e43ea2117230c9: Status 404 returned error can't find the container with id 04e2892022ae4054c8c7bc861ac2de58728d493f209afaf3d5e43ea2117230c9
Jul 17 22:42:57.964005 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.963898    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q" event={"ID":"ed6c2901-7a95-49d9-aba6-5315349636ab","Type":"ContainerStarted","Data":"839513a1c69e19bc71872636d78d54c9c804c57cabd036148b8310cb00515f07"}
Jul 17 22:42:57.964005 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.963965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q" event={"ID":"ed6c2901-7a95-49d9-aba6-5315349636ab","Type":"ContainerStarted","Data":"04e2892022ae4054c8c7bc861ac2de58728d493f209afaf3d5e43ea2117230c9"}
Jul 17 22:42:57.965685 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.965659    2565 generic.go:356] "Generic (PLEG): container finished" podID="c074c4d3-ba15-4058-a0b4-812e4e543e10" containerID="f1a86f053b82b4cd2d98fdcab7b21e7b874bdc180a48985b26cdd8d744723869" exitCode=6
Jul 17 22:42:57.965788 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.965699    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs" event={"ID":"c074c4d3-ba15-4058-a0b4-812e4e543e10","Type":"ContainerDied","Data":"f1a86f053b82b4cd2d98fdcab7b21e7b874bdc180a48985b26cdd8d744723869"}
Jul 17 22:42:57.965788 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.965744    2565 scope.go:117] "RemoveContainer" containerID="ec70cd3efb6b4798f672384fb042f172d9f67e88d1a2ecc6a1cae9bccd45d873"
Jul 17 22:42:57.965902 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.965887    2565 scope.go:117] "RemoveContainer" containerID="f1a86f053b82b4cd2d98fdcab7b21e7b874bdc180a48985b26cdd8d744723869"
Jul 17 22:42:57.966127 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:42:57.966107    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs_odh-ai-gateway-infra(c074c4d3-ba15-4058-a0b4-812e4e543e10)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs" podUID="c074c4d3-ba15-4058-a0b4-812e4e543e10"
Jul 17 22:42:57.982865 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:57.982823    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q" podStartSLOduration=0.982809193 podStartE2EDuration="982.809193ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:42:57.980299853 +0000 UTC m=+3441.369316505" watchObservedRunningTime="2026-07-17 22:42:57.982809193 +0000 UTC m=+3441.371825843"
Jul 17 22:42:58.071149 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:58.071110    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:42:58.972159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:58.972130    2565 generic.go:356] "Generic (PLEG): container finished" podID="ed6c2901-7a95-49d9-aba6-5315349636ab" containerID="839513a1c69e19bc71872636d78d54c9c804c57cabd036148b8310cb00515f07" exitCode=6
Jul 17 22:42:58.972589 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:58.972213    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q" event={"ID":"ed6c2901-7a95-49d9-aba6-5315349636ab","Type":"ContainerDied","Data":"839513a1c69e19bc71872636d78d54c9c804c57cabd036148b8310cb00515f07"}
Jul 17 22:42:58.972589 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:58.972449    2565 scope.go:117] "RemoveContainer" containerID="839513a1c69e19bc71872636d78d54c9c804c57cabd036148b8310cb00515f07"
Jul 17 22:42:58.974302 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:58.974284    2565 scope.go:117] "RemoveContainer" containerID="f1a86f053b82b4cd2d98fdcab7b21e7b874bdc180a48985b26cdd8d744723869"
Jul 17 22:42:58.974455 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:42:58.974439    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs_odh-ai-gateway-infra(c074c4d3-ba15-4058-a0b4-812e4e543e10)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs" podUID="c074c4d3-ba15-4058-a0b4-812e4e543e10"
Jul 17 22:42:59.982384 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:59.982291    2565 generic.go:356] "Generic (PLEG): container finished" podID="ed6c2901-7a95-49d9-aba6-5315349636ab" containerID="64217544c8eda6a537338d2a70af9115dd67fc7a6487d6720e232297896c0f9d" exitCode=6
Jul 17 22:42:59.982782 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:59.982380    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q" event={"ID":"ed6c2901-7a95-49d9-aba6-5315349636ab","Type":"ContainerDied","Data":"64217544c8eda6a537338d2a70af9115dd67fc7a6487d6720e232297896c0f9d"}
Jul 17 22:42:59.982782 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:59.982449    2565 scope.go:117] "RemoveContainer" containerID="839513a1c69e19bc71872636d78d54c9c804c57cabd036148b8310cb00515f07"
Jul 17 22:42:59.982782 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:42:59.982695    2565 scope.go:117] "RemoveContainer" containerID="64217544c8eda6a537338d2a70af9115dd67fc7a6487d6720e232297896c0f9d"
Jul 17 22:42:59.982936 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:42:59.982914    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q_odh-ai-gateway-infra(ed6c2901-7a95-49d9-aba6-5315349636ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q" podUID="ed6c2901-7a95-49d9-aba6-5315349636ab"
Jul 17 22:43:00.989141 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:00.989113    2565 scope.go:117] "RemoveContainer" containerID="64217544c8eda6a537338d2a70af9115dd67fc7a6487d6720e232297896c0f9d"
Jul 17 22:43:00.989610 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:43:00.989333    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q_odh-ai-gateway-infra(ed6c2901-7a95-49d9-aba6-5315349636ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q" podUID="ed6c2901-7a95-49d9-aba6-5315349636ab"
Jul 17 22:43:04.222454 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:04.222419    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:43:04.292831 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:04.292795    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:43:08.739691 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:08.739647    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"]
Jul 17 22:43:08.740143 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:08.739919    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd" podUID="df489087-22f5-4d73-878b-1d5deed40d78" containerName="istio-proxy" containerID="cri-o://4176535d4c81fdd935bab80cf982eff4bf412f9892f82d702145b1c04eb6d253" gracePeriod=30
Jul 17 22:43:08.765043 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:08.765009    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:43:08.990516 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:08.990432    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:43:09.853309 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:09.853267    2565 patch_prober.go:28] interesting pod/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 22:43:09.853706 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:09.853337    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd" podUID="df489087-22f5-4d73-878b-1d5deed40d78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 22:43:12.177318 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:12.177285    2565 scope.go:117] "RemoveContainer" containerID="64217544c8eda6a537338d2a70af9115dd67fc7a6487d6720e232297896c0f9d"
Jul 17 22:43:13.049749 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:13.049659    2565 generic.go:356] "Generic (PLEG): container finished" podID="ed6c2901-7a95-49d9-aba6-5315349636ab" containerID="609db18a5878ecedea1332d9198e4895214bd15aa3b4a3e545f156ed5b6be0cb" exitCode=6
Jul 17 22:43:13.049749 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:13.049700    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q" event={"ID":"ed6c2901-7a95-49d9-aba6-5315349636ab","Type":"ContainerDied","Data":"609db18a5878ecedea1332d9198e4895214bd15aa3b4a3e545f156ed5b6be0cb"}
Jul 17 22:43:13.049749 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:13.049748    2565 scope.go:117] "RemoveContainer" containerID="64217544c8eda6a537338d2a70af9115dd67fc7a6487d6720e232297896c0f9d"
Jul 17 22:43:13.050113 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:13.050094    2565 scope.go:117] "RemoveContainer" containerID="609db18a5878ecedea1332d9198e4895214bd15aa3b4a3e545f156ed5b6be0cb"
Jul 17 22:43:13.050361 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:43:13.050339    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q_odh-ai-gateway-infra(ed6c2901-7a95-49d9-aba6-5315349636ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q" podUID="ed6c2901-7a95-49d9-aba6-5315349636ab"
Jul 17 22:43:13.494331 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:13.494240    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6"]
Jul 17 22:43:13.498924 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:13.498907    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6"
Jul 17 22:43:13.504788 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:13.504703    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6"]
Jul 17 22:43:13.557522 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:13.557490    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e634b5ee-b530-4c81-afb1-15f718eaace2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6\" (UID: \"e634b5ee-b530-4c81-afb1-15f718eaace2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6"
Jul 17 22:43:13.659016 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:13.658953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e634b5ee-b530-4c81-afb1-15f718eaace2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6\" (UID: \"e634b5ee-b530-4c81-afb1-15f718eaace2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6"
Jul 17 22:43:13.659673 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:13.659652    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e634b5ee-b530-4c81-afb1-15f718eaace2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6\" (UID: \"e634b5ee-b530-4c81-afb1-15f718eaace2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6"
Jul 17 22:43:13.811951 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:13.811917    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6"
Jul 17 22:43:14.020182 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.020156    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:43:14.056950 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.056915    2565 generic.go:356] "Generic (PLEG): container finished" podID="df489087-22f5-4d73-878b-1d5deed40d78" containerID="4176535d4c81fdd935bab80cf982eff4bf412f9892f82d702145b1c04eb6d253" exitCode=0
Jul 17 22:43:14.057183 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.056974    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"
Jul 17 22:43:14.057183 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.056989    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd" event={"ID":"df489087-22f5-4d73-878b-1d5deed40d78","Type":"ContainerDied","Data":"4176535d4c81fdd935bab80cf982eff4bf412f9892f82d702145b1c04eb6d253"}
Jul 17 22:43:14.057183 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.057053    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd" event={"ID":"df489087-22f5-4d73-878b-1d5deed40d78","Type":"ContainerDied","Data":"0dec833eebfff8c124da6dfb6c41edab0688342520d9ea9a91bc2b0edf91b315"}
Jul 17 22:43:14.057183 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.057107    2565 scope.go:117] "RemoveContainer" containerID="4176535d4c81fdd935bab80cf982eff4bf412f9892f82d702145b1c04eb6d253"
Jul 17 22:43:14.062702 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.062678    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnf6v\" (UniqueName: \"kubernetes.io/projected/df489087-22f5-4d73-878b-1d5deed40d78-kube-api-access-wnf6v\") pod \"df489087-22f5-4d73-878b-1d5deed40d78\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") "
Jul 17 22:43:14.062813 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.062716    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-workload-certs\") pod \"df489087-22f5-4d73-878b-1d5deed40d78\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") "
Jul 17 22:43:14.062813 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.062767    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-istio-envoy\") pod \"df489087-22f5-4d73-878b-1d5deed40d78\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") "
Jul 17 22:43:14.062939 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.062819    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-credential-socket\") pod \"df489087-22f5-4d73-878b-1d5deed40d78\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") "
Jul 17 22:43:14.062939 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.062865    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df489087-22f5-4d73-878b-1d5deed40d78-istio-podinfo\") pod \"df489087-22f5-4d73-878b-1d5deed40d78\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") "
Jul 17 22:43:14.062939 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.062899    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df489087-22f5-4d73-878b-1d5deed40d78-istio-token\") pod \"df489087-22f5-4d73-878b-1d5deed40d78\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") "
Jul 17 22:43:14.062939 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.062935    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df489087-22f5-4d73-878b-1d5deed40d78-istiod-ca-cert\") pod \"df489087-22f5-4d73-878b-1d5deed40d78\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") "
Jul 17 22:43:14.063190 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.062973    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-istio-data\") pod \"df489087-22f5-4d73-878b-1d5deed40d78\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") "
Jul 17 22:43:14.063190 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.063017    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-workload-socket\") pod \"df489087-22f5-4d73-878b-1d5deed40d78\" (UID: \"df489087-22f5-4d73-878b-1d5deed40d78\") "
Jul 17 22:43:14.063190 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.063115    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "df489087-22f5-4d73-878b-1d5deed40d78" (UID: "df489087-22f5-4d73-878b-1d5deed40d78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:43:14.063591 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.063368    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-workload-certs\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:43:14.063591 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.063430    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df489087-22f5-4d73-878b-1d5deed40d78-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "df489087-22f5-4d73-878b-1d5deed40d78" (UID: "df489087-22f5-4d73-878b-1d5deed40d78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:43:14.063948 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.063760    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "df489087-22f5-4d73-878b-1d5deed40d78" (UID: "df489087-22f5-4d73-878b-1d5deed40d78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:43:14.064128 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.064094    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "df489087-22f5-4d73-878b-1d5deed40d78" (UID: "df489087-22f5-4d73-878b-1d5deed40d78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:43:14.064215 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.064126    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-istio-data" (OuterVolumeSpecName: "istio-data") pod "df489087-22f5-4d73-878b-1d5deed40d78" (UID: "df489087-22f5-4d73-878b-1d5deed40d78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:43:14.065448 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.065426    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df489087-22f5-4d73-878b-1d5deed40d78-kube-api-access-wnf6v" (OuterVolumeSpecName: "kube-api-access-wnf6v") pod "df489087-22f5-4d73-878b-1d5deed40d78" (UID: "df489087-22f5-4d73-878b-1d5deed40d78"). InnerVolumeSpecName "kube-api-access-wnf6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:43:14.065541 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.065442    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df489087-22f5-4d73-878b-1d5deed40d78-istio-token" (OuterVolumeSpecName: "istio-token") pod "df489087-22f5-4d73-878b-1d5deed40d78" (UID: "df489087-22f5-4d73-878b-1d5deed40d78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:43:14.065733 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.065714    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "df489087-22f5-4d73-878b-1d5deed40d78" (UID: "df489087-22f5-4d73-878b-1d5deed40d78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:43:14.065923 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.065906    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df489087-22f5-4d73-878b-1d5deed40d78-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "df489087-22f5-4d73-878b-1d5deed40d78" (UID: "df489087-22f5-4d73-878b-1d5deed40d78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:43:14.067782 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.067760    2565 scope.go:117] "RemoveContainer" containerID="d5cd36cab17e2923fde35943d9ba1658ccd91c3a146c75c6e87ece6caef15241"
Jul 17 22:43:14.075504 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.075476    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q"]
Jul 17 22:43:14.076966 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.076945    2565 scope.go:117] "RemoveContainer" containerID="4176535d4c81fdd935bab80cf982eff4bf412f9892f82d702145b1c04eb6d253"
Jul 17 22:43:14.077380 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:43:14.077357    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4176535d4c81fdd935bab80cf982eff4bf412f9892f82d702145b1c04eb6d253\": container with ID starting with 4176535d4c81fdd935bab80cf982eff4bf412f9892f82d702145b1c04eb6d253 not found: ID does not exist" containerID="4176535d4c81fdd935bab80cf982eff4bf412f9892f82d702145b1c04eb6d253"
Jul 17 22:43:14.077443 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.077391    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4176535d4c81fdd935bab80cf982eff4bf412f9892f82d702145b1c04eb6d253"} err="failed to get container status \"4176535d4c81fdd935bab80cf982eff4bf412f9892f82d702145b1c04eb6d253\": rpc error: code = NotFound desc = could not find container \"4176535d4c81fdd935bab80cf982eff4bf412f9892f82d702145b1c04eb6d253\": container with ID starting with 4176535d4c81fdd935bab80cf982eff4bf412f9892f82d702145b1c04eb6d253 not found: ID does not exist"
Jul 17 22:43:14.077443 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.077417    2565 scope.go:117] "RemoveContainer" containerID="d5cd36cab17e2923fde35943d9ba1658ccd91c3a146c75c6e87ece6caef15241"
Jul 17 22:43:14.077669 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:43:14.077653    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5cd36cab17e2923fde35943d9ba1658ccd91c3a146c75c6e87ece6caef15241\": container with ID starting with d5cd36cab17e2923fde35943d9ba1658ccd91c3a146c75c6e87ece6caef15241 not found: ID does not exist" containerID="d5cd36cab17e2923fde35943d9ba1658ccd91c3a146c75c6e87ece6caef15241"
Jul 17 22:43:14.077712 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.077676    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5cd36cab17e2923fde35943d9ba1658ccd91c3a146c75c6e87ece6caef15241"} err="failed to get container status \"d5cd36cab17e2923fde35943d9ba1658ccd91c3a146c75c6e87ece6caef15241\": rpc error: code = NotFound desc = could not find container \"d5cd36cab17e2923fde35943d9ba1658ccd91c3a146c75c6e87ece6caef15241\": container with ID starting with d5cd36cab17e2923fde35943d9ba1658ccd91c3a146c75c6e87ece6caef15241 not found: ID does not exist"
Jul 17 22:43:14.164457 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.164422    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df489087-22f5-4d73-878b-1d5deed40d78-istio-token\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:43:14.164457 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.164459    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df489087-22f5-4d73-878b-1d5deed40d78-istiod-ca-cert\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:43:14.164673 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.164473    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-istio-data\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:43:14.164673 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.164486    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-workload-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:43:14.164673 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.164498    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnf6v\" (UniqueName: \"kubernetes.io/projected/df489087-22f5-4d73-878b-1d5deed40d78-kube-api-access-wnf6v\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:43:14.164673 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.164510    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-istio-envoy\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:43:14.164673 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.164521    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df489087-22f5-4d73-878b-1d5deed40d78-credential-socket\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:43:14.164673 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.164532    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df489087-22f5-4d73-878b-1d5deed40d78-istio-podinfo\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:43:14.166834 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.166805    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6"]
Jul 17 22:43:14.168034 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:43:14.168011    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode634b5ee_b530_4c81_afb1_15f718eaace2.slice/crio-763a3a4dc6fd2d2ea73e4eadbb4c1ef9292d611fcb2eeff57a746d99b2dfa8b0 WatchSource:0}: Error finding container 763a3a4dc6fd2d2ea73e4eadbb4c1ef9292d611fcb2eeff57a746d99b2dfa8b0: Status 404 returned error can't find the container with id 763a3a4dc6fd2d2ea73e4eadbb4c1ef9292d611fcb2eeff57a746d99b2dfa8b0
Jul 17 22:43:14.177390 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.177369    2565 scope.go:117] "RemoveContainer" containerID="f1a86f053b82b4cd2d98fdcab7b21e7b874bdc180a48985b26cdd8d744723869"
Jul 17 22:43:14.219375 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.219348    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q"
Jul 17 22:43:14.265353 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.265262    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6c2901-7a95-49d9-aba6-5315349636ab-maas-api-service-ca\") pod \"ed6c2901-7a95-49d9-aba6-5315349636ab\" (UID: \"ed6c2901-7a95-49d9-aba6-5315349636ab\") "
Jul 17 22:43:14.265626 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.265597    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6c2901-7a95-49d9-aba6-5315349636ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed6c2901-7a95-49d9-aba6-5315349636ab" (UID: "ed6c2901-7a95-49d9-aba6-5315349636ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:43:14.366624 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.366598    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6c2901-7a95-49d9-aba6-5315349636ab-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:43:14.389913 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.389880    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"]
Jul 17 22:43:14.394359 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.394332    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-fb05ec-openshift-default-7fcb4bdc9-mwgqd"]
Jul 17 22:43:14.814596 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.814520    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-r56kt_1416461e-cf9d-4168-aca9-b496c64bcf7c/manager/0.log"
Jul 17 22:43:14.933548 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:14.933519    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5fl2x_d2089a42-8dbb-413a-b6cf-749899b408c0/manager/0.log"
Jul 17 22:43:15.064935 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.064846    2565 generic.go:356] "Generic (PLEG): container finished" podID="c074c4d3-ba15-4058-a0b4-812e4e543e10" containerID="f2e74c0073909faf473b8dd9bb008eae277c8e5ab6a6f86ec6ed6e1f3ecff39b" exitCode=6
Jul 17 22:43:15.064935 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.064925    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs" event={"ID":"c074c4d3-ba15-4058-a0b4-812e4e543e10","Type":"ContainerDied","Data":"f2e74c0073909faf473b8dd9bb008eae277c8e5ab6a6f86ec6ed6e1f3ecff39b"}
Jul 17 22:43:15.065159 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.064986    2565 scope.go:117] "RemoveContainer" containerID="f1a86f053b82b4cd2d98fdcab7b21e7b874bdc180a48985b26cdd8d744723869"
Jul 17 22:43:15.065367 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.065337    2565 scope.go:117] "RemoveContainer" containerID="f2e74c0073909faf473b8dd9bb008eae277c8e5ab6a6f86ec6ed6e1f3ecff39b"
Jul 17 22:43:15.065626 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:43:15.065606    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs_odh-ai-gateway-infra(c074c4d3-ba15-4058-a0b4-812e4e543e10)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs" podUID="c074c4d3-ba15-4058-a0b4-812e4e543e10"
Jul 17 22:43:15.066485 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.066465    2565 generic.go:356] "Generic (PLEG): container finished" podID="e634b5ee-b530-4c81-afb1-15f718eaace2" containerID="79dc2f32b7c314c5621ce34ded41f17c6327e09378747e2e099ac15a8fe2bf02" exitCode=6
Jul 17 22:43:15.066574 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.066551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6" event={"ID":"e634b5ee-b530-4c81-afb1-15f718eaace2","Type":"ContainerDied","Data":"79dc2f32b7c314c5621ce34ded41f17c6327e09378747e2e099ac15a8fe2bf02"}
Jul 17 22:43:15.066635 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.066604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6" event={"ID":"e634b5ee-b530-4c81-afb1-15f718eaace2","Type":"ContainerStarted","Data":"763a3a4dc6fd2d2ea73e4eadbb4c1ef9292d611fcb2eeff57a746d99b2dfa8b0"}
Jul 17 22:43:15.066716 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.066699    2565 scope.go:117] "RemoveContainer" containerID="79dc2f32b7c314c5621ce34ded41f17c6327e09378747e2e099ac15a8fe2bf02"
Jul 17 22:43:15.068711 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.068690    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q" event={"ID":"ed6c2901-7a95-49d9-aba6-5315349636ab","Type":"ContainerDied","Data":"04e2892022ae4054c8c7bc861ac2de58728d493f209afaf3d5e43ea2117230c9"}
Jul 17 22:43:15.068821 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.068703    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q"
Jul 17 22:43:15.079559 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.079538    2565 scope.go:117] "RemoveContainer" containerID="609db18a5878ecedea1332d9198e4895214bd15aa3b4a3e545f156ed5b6be0cb"
Jul 17 22:43:15.103014 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.102987    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q"]
Jul 17 22:43:15.107282 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.107247    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-5kg7q"]
Jul 17 22:43:15.182456 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.182423    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df489087-22f5-4d73-878b-1d5deed40d78" path="/var/lib/kubelet/pods/df489087-22f5-4d73-878b-1d5deed40d78/volumes"
Jul 17 22:43:15.182769 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.182756    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed6c2901-7a95-49d9-aba6-5315349636ab" path="/var/lib/kubelet/pods/ed6c2901-7a95-49d9-aba6-5315349636ab/volumes"
Jul 17 22:43:15.278841 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.278816    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-z8r8m_877e02a7-c53c-4331-b7d0-409aa961ec39/manager/0.log"
Jul 17 22:43:15.647292 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.647214    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bff9999cf-xpc64_eec8dce4-acea-42e6-add7-879b08ce7aa8/copy-manifests/0.log"
Jul 17 22:43:15.654447 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:15.654420    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bff9999cf-xpc64_eec8dce4-acea-42e6-add7-879b08ce7aa8/manager/0.log"
Jul 17 22:43:16.084053 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:16.084019    2565 generic.go:356] "Generic (PLEG): container finished" podID="e634b5ee-b530-4c81-afb1-15f718eaace2" containerID="c8b7058113a8962ddd40b186571b30ce60a67e370c65478c43aafdee2488827f" exitCode=6
Jul 17 22:43:16.084053 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:16.084059    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6" event={"ID":"e634b5ee-b530-4c81-afb1-15f718eaace2","Type":"ContainerDied","Data":"c8b7058113a8962ddd40b186571b30ce60a67e370c65478c43aafdee2488827f"}
Jul 17 22:43:16.084617 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:16.084148    2565 scope.go:117] "RemoveContainer" containerID="79dc2f32b7c314c5621ce34ded41f17c6327e09378747e2e099ac15a8fe2bf02"
Jul 17 22:43:16.084617 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:16.084299    2565 scope.go:117] "RemoveContainer" containerID="c8b7058113a8962ddd40b186571b30ce60a67e370c65478c43aafdee2488827f"
Jul 17 22:43:16.084617 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:43:16.084534    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6_odh-ai-gateway-infra(e634b5ee-b530-4c81-afb1-15f718eaace2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6" podUID="e634b5ee-b530-4c81-afb1-15f718eaace2"
Jul 17 22:43:16.098012 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:16.097974    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs"]
Jul 17 22:43:16.135919 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:16.135894    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738760-4d9n7_f84846b0-0ae6-448d-9670-1d066562faa9/cleanup/0.log"
Jul 17 22:43:16.242098 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:16.242054    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs"
Jul 17 22:43:16.249920 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:16.249891    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738775-lbrm2_053efeeb-991e-4e1d-9146-4e3872cb90c8/cleanup/0.log"
Jul 17 22:43:16.282197 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:16.282163    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c074c4d3-ba15-4058-a0b4-812e4e543e10-maas-api-service-ca\") pod \"c074c4d3-ba15-4058-a0b4-812e4e543e10\" (UID: \"c074c4d3-ba15-4058-a0b4-812e4e543e10\") "
Jul 17 22:43:16.282541 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:16.282521    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c074c4d3-ba15-4058-a0b4-812e4e543e10-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c074c4d3-ba15-4058-a0b4-812e4e543e10" (UID: "c074c4d3-ba15-4058-a0b4-812e4e543e10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:43:16.360671 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:16.360592    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738790-7xgkf_d051730a-986a-4be3-befe-03a1470b857d/cleanup/0.log"
Jul 17 22:43:16.383116 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:16.383050    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c074c4d3-ba15-4058-a0b4-812e4e543e10-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:43:16.465647 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:16.465614    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs_c074c4d3-ba15-4058-a0b4-812e4e543e10/revoke-keys/2.log"
Jul 17 22:43:16.808744 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:16.808713    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6_e634b5ee-b530-4c81-afb1-15f718eaace2/revoke-keys/1.log"
Jul 17 22:43:16.924707 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:16.924669    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-11aecf-fkbvs_4e2efb5d-9490-43af-ba4d-9e5cde98129e/revoke-keys/0.log"
Jul 17 22:43:17.091231 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:17.091114    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs"
Jul 17 22:43:17.091231 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:17.091131    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs" event={"ID":"c074c4d3-ba15-4058-a0b4-812e4e543e10","Type":"ContainerDied","Data":"edfd26e142c950f34d436830776d86b099f20fdf9df867058e5889fae26284bb"}
Jul 17 22:43:17.092141 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:17.091293    2565 scope.go:117] "RemoveContainer" containerID="f2e74c0073909faf473b8dd9bb008eae277c8e5ab6a6f86ec6ed6e1f3ecff39b"
Jul 17 22:43:17.093662 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:17.093640    2565 scope.go:117] "RemoveContainer" containerID="c8b7058113a8962ddd40b186571b30ce60a67e370c65478c43aafdee2488827f"
Jul 17 22:43:17.093935 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:43:17.093898    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6_odh-ai-gateway-infra(e634b5ee-b530-4c81-afb1-15f718eaace2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6" podUID="e634b5ee-b530-4c81-afb1-15f718eaace2"
Jul 17 22:43:17.124831 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:17.124794    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs"]
Jul 17 22:43:17.130744 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:17.130714    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-vdtqs"]
Jul 17 22:43:17.147462 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:17.147434    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-895e50b8-nj754_69f21a49-3718-4779-b42b-938bf462ba74/revoke-keys/0.log"
Jul 17 22:43:17.182137 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:17.182095    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c074c4d3-ba15-4058-a0b4-812e4e543e10" path="/var/lib/kubelet/pods/c074c4d3-ba15-4058-a0b4-812e4e543e10/volumes"
Jul 17 22:43:17.262929 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:17.262898    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-c8731528-ztj4x_d80a957c-30ac-44e2-a1dc-f8d233b8a189/revoke-keys/0.log"
Jul 17 22:43:19.622834 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:19.622807    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-2g44p_8f5ee100-6729-445b-ac7d-60d590c960cc/limitador/0.log"
Jul 17 22:43:19.737335 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:19.737304    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-sqgcc_5d5837b7-e157-4a0e-836a-fab4f0bd6ddf/manager/0.log"
Jul 17 22:43:20.093887 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:20.093855    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6782hq7p_314b915f-30de-4d8d-af9e-01f8d2b7fd08/istio-proxy/0.log"
Jul 17 22:43:20.213326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:20.213295    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-8l8sd_b33878cc-2675-4a04-ad0d-7a2cf0d4db99/discovery/0.log"
Jul 17 22:43:20.435422 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:20.435345    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85fdd95b7f-x9zvr_81716365-9c9f-4819-b6d4-e948453cb094/kube-auth-proxy/0.log"
Jul 17 22:43:20.554568 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:20.554532    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-pvfz2_fc686a0b-25d0-4d64-bf27-ff5ee005090b/istio-proxy/1.log"
Jul 17 22:43:21.579014 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:21.578980    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvpx65_82ec16d0-bf9c-4708-be9b-86209944c32b/main/0.log"
Jul 17 22:43:25.540204 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:25.540168    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:43:26.790909 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:26.790871    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:43:27.814319 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:27.814283    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:43:27.889377 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:27.889340    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log" limit=1024
Jul 17 22:43:28.176878 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.176783    2565 scope.go:117] "RemoveContainer" containerID="c8b7058113a8962ddd40b186571b30ce60a67e370c65478c43aafdee2488827f"
Jul 17 22:43:28.876509 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876478    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q"]
Jul 17 22:43:28.876941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876823    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c074c4d3-ba15-4058-a0b4-812e4e543e10" containerName="revoke-keys"
Jul 17 22:43:28.876941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876833    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c074c4d3-ba15-4058-a0b4-812e4e543e10" containerName="revoke-keys"
Jul 17 22:43:28.876941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876843    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df489087-22f5-4d73-878b-1d5deed40d78" containerName="istio-proxy"
Jul 17 22:43:28.876941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876849    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="df489087-22f5-4d73-878b-1d5deed40d78" containerName="istio-proxy"
Jul 17 22:43:28.876941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876855    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df489087-22f5-4d73-878b-1d5deed40d78" containerName="istio-proxy"
Jul 17 22:43:28.876941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876861    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="df489087-22f5-4d73-878b-1d5deed40d78" containerName="istio-proxy"
Jul 17 22:43:28.876941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876873    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6c2901-7a95-49d9-aba6-5315349636ab" containerName="revoke-keys"
Jul 17 22:43:28.876941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876879    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6c2901-7a95-49d9-aba6-5315349636ab" containerName="revoke-keys"
Jul 17 22:43:28.876941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876890    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6c2901-7a95-49d9-aba6-5315349636ab" containerName="revoke-keys"
Jul 17 22:43:28.876941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876895    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6c2901-7a95-49d9-aba6-5315349636ab" containerName="revoke-keys"
Jul 17 22:43:28.876941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876904    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6c2901-7a95-49d9-aba6-5315349636ab" containerName="revoke-keys"
Jul 17 22:43:28.876941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876909    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6c2901-7a95-49d9-aba6-5315349636ab" containerName="revoke-keys"
Jul 17 22:43:28.876941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876916    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c074c4d3-ba15-4058-a0b4-812e4e543e10" containerName="revoke-keys"
Jul 17 22:43:28.876941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876922    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c074c4d3-ba15-4058-a0b4-812e4e543e10" containerName="revoke-keys"
Jul 17 22:43:28.877664 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876969    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed6c2901-7a95-49d9-aba6-5315349636ab" containerName="revoke-keys"
Jul 17 22:43:28.877664 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876978    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c074c4d3-ba15-4058-a0b4-812e4e543e10" containerName="revoke-keys"
Jul 17 22:43:28.877664 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876984    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="df489087-22f5-4d73-878b-1d5deed40d78" containerName="istio-proxy"
Jul 17 22:43:28.877664 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876989    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="df489087-22f5-4d73-878b-1d5deed40d78" containerName="istio-proxy"
Jul 17 22:43:28.877664 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.876996    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed6c2901-7a95-49d9-aba6-5315349636ab" containerName="revoke-keys"
Jul 17 22:43:28.877664 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.877001    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c074c4d3-ba15-4058-a0b4-812e4e543e10" containerName="revoke-keys"
Jul 17 22:43:28.877664 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.877006    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c074c4d3-ba15-4058-a0b4-812e4e543e10" containerName="revoke-keys"
Jul 17 22:43:28.879937 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.879918    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q"
Jul 17 22:43:28.889999 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.889972    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q"]
Jul 17 22:43:28.993990 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:28.993957    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19035086-3bce-4efb-b040-d5305b4d324f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q\" (UID: \"19035086-3bce-4efb-b040-d5305b4d324f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q"
Jul 17 22:43:29.094569 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:29.094534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19035086-3bce-4efb-b040-d5305b4d324f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q\" (UID: \"19035086-3bce-4efb-b040-d5305b4d324f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q"
Jul 17 22:43:29.095210 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:29.095190    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19035086-3bce-4efb-b040-d5305b4d324f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q\" (UID: \"19035086-3bce-4efb-b040-d5305b4d324f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q"
Jul 17 22:43:29.154405 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:29.154306    2565 generic.go:356] "Generic (PLEG): container finished" podID="e634b5ee-b530-4c81-afb1-15f718eaace2" containerID="54af6b0aaac9ff6c60f90e210bd3300689852d73722f7efbbe1b3612b1192522" exitCode=6
Jul 17 22:43:29.154405 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:29.154384    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6" event={"ID":"e634b5ee-b530-4c81-afb1-15f718eaace2","Type":"ContainerDied","Data":"54af6b0aaac9ff6c60f90e210bd3300689852d73722f7efbbe1b3612b1192522"}
Jul 17 22:43:29.154609 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:29.154477    2565 scope.go:117] "RemoveContainer" containerID="c8b7058113a8962ddd40b186571b30ce60a67e370c65478c43aafdee2488827f"
Jul 17 22:43:29.154852 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:29.154832    2565 scope.go:117] "RemoveContainer" containerID="54af6b0aaac9ff6c60f90e210bd3300689852d73722f7efbbe1b3612b1192522"
Jul 17 22:43:29.155079 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:43:29.155039    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6_odh-ai-gateway-infra(e634b5ee-b530-4c81-afb1-15f718eaace2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6" podUID="e634b5ee-b530-4c81-afb1-15f718eaace2"
Jul 17 22:43:29.234421 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:29.234390    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q"
Jul 17 22:43:29.375081 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:29.375036    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q"]
Jul 17 22:43:29.377572 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:43:29.377538    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19035086_3bce_4efb_b040_d5305b4d324f.slice/crio-0a6921fd3c35b784041b5f8453b533e94cfa2ef01b92de3d36b8e1623af39980 WatchSource:0}: Error finding container 0a6921fd3c35b784041b5f8453b533e94cfa2ef01b92de3d36b8e1623af39980: Status 404 returned error can't find the container with id 0a6921fd3c35b784041b5f8453b533e94cfa2ef01b92de3d36b8e1623af39980
Jul 17 22:43:30.163979 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:30.163936    2565 generic.go:356] "Generic (PLEG): container finished" podID="19035086-3bce-4efb-b040-d5305b4d324f" containerID="a080a91a72259896a8e3639b151db8f3b8ef7715c28459f956ecd72e52b63d61" exitCode=6
Jul 17 22:43:30.164429 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:30.164023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q" event={"ID":"19035086-3bce-4efb-b040-d5305b4d324f","Type":"ContainerDied","Data":"a080a91a72259896a8e3639b151db8f3b8ef7715c28459f956ecd72e52b63d61"}
Jul 17 22:43:30.164429 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:30.164089    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q" event={"ID":"19035086-3bce-4efb-b040-d5305b4d324f","Type":"ContainerStarted","Data":"0a6921fd3c35b784041b5f8453b533e94cfa2ef01b92de3d36b8e1623af39980"}
Jul 17 22:43:30.164429 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:30.164194    2565 scope.go:117] "RemoveContainer" containerID="a080a91a72259896a8e3639b151db8f3b8ef7715c28459f956ecd72e52b63d61"
Jul 17 22:43:30.190305 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:30.190276    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6"]
Jul 17 22:43:30.330674 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:30.330647    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6"
Jul 17 22:43:30.406721 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:30.406690    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e634b5ee-b530-4c81-afb1-15f718eaace2-maas-api-service-ca\") pod \"e634b5ee-b530-4c81-afb1-15f718eaace2\" (UID: \"e634b5ee-b530-4c81-afb1-15f718eaace2\") "
Jul 17 22:43:30.407135 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:30.407111    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e634b5ee-b530-4c81-afb1-15f718eaace2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e634b5ee-b530-4c81-afb1-15f718eaace2" (UID: "e634b5ee-b530-4c81-afb1-15f718eaace2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:43:30.507449 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:30.507358    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e634b5ee-b530-4c81-afb1-15f718eaace2-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:43:31.012037 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:31.011930    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hf7sc_aeb396b3-2508-4c93-aeee-188c1b39fa9d/global-pull-secret-syncer/0.log"
Jul 17 22:43:31.113805 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:31.113774    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5xm2r_00ccd718-0d99-4948-84b4-6d0aaff1598b/konnectivity-agent/0.log"
Jul 17 22:43:31.170732 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:31.170690    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6" event={"ID":"e634b5ee-b530-4c81-afb1-15f718eaace2","Type":"ContainerDied","Data":"763a3a4dc6fd2d2ea73e4eadbb4c1ef9292d611fcb2eeff57a746d99b2dfa8b0"}
Jul 17 22:43:31.171228 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:31.170755    2565 scope.go:117] "RemoveContainer" containerID="54af6b0aaac9ff6c60f90e210bd3300689852d73722f7efbbe1b3612b1192522"
Jul 17 22:43:31.171228 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:31.170699    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6"
Jul 17 22:43:31.172670 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:31.172646    2565 generic.go:356] "Generic (PLEG): container finished" podID="19035086-3bce-4efb-b040-d5305b4d324f" containerID="1b7656c5c97b7667921d70e796d81430b09a45020263dfcd98e1a72e8450dec1" exitCode=6
Jul 17 22:43:31.172789 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:31.172687    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q" event={"ID":"19035086-3bce-4efb-b040-d5305b4d324f","Type":"ContainerDied","Data":"1b7656c5c97b7667921d70e796d81430b09a45020263dfcd98e1a72e8450dec1"}
Jul 17 22:43:31.172968 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:31.172951    2565 scope.go:117] "RemoveContainer" containerID="1b7656c5c97b7667921d70e796d81430b09a45020263dfcd98e1a72e8450dec1"
Jul 17 22:43:31.173217 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:43:31.173197    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q_odh-ai-gateway-infra(19035086-3bce-4efb-b040-d5305b4d324f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q" podUID="19035086-3bce-4efb-b040-d5305b4d324f"
Jul 17 22:43:31.180661 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:31.180643    2565 scope.go:117] "RemoveContainer" containerID="a080a91a72259896a8e3639b151db8f3b8ef7715c28459f956ecd72e52b63d61"
Jul 17 22:43:31.205624 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:31.205584    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6"]
Jul 17 22:43:31.211844 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:31.211816    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-11.ec2.internal_1f1502cc83beae535f93cdbb53034f5e/haproxy/0.log"
Jul 17 22:43:31.212436 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:31.212416    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-4vzk6"]
Jul 17 22:43:32.180715 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:32.180684    2565 scope.go:117] "RemoveContainer" containerID="1b7656c5c97b7667921d70e796d81430b09a45020263dfcd98e1a72e8450dec1"
Jul 17 22:43:32.181140 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:43:32.180933    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q_odh-ai-gateway-infra(19035086-3bce-4efb-b040-d5305b4d324f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q" podUID="19035086-3bce-4efb-b040-d5305b4d324f"
Jul 17 22:43:33.182888 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:33.182848    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e634b5ee-b530-4c81-afb1-15f718eaace2" path="/var/lib/kubelet/pods/e634b5ee-b530-4c81-afb1-15f718eaace2/volumes"
Jul 17 22:43:36.713168 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:36.713139    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-2g44p_8f5ee100-6729-445b-ac7d-60d590c960cc/limitador/0.log"
Jul 17 22:43:36.778326 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:36.778288    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-sqgcc_5d5837b7-e157-4a0e-836a-fab4f0bd6ddf/manager/0.log"
Jul 17 22:43:38.879219 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:38.879190    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zq9m4_48f183d4-25b4-4f70-a9df-b82f914e7b58/node-exporter/0.log"
Jul 17 22:43:38.909039 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:38.909012    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zq9m4_48f183d4-25b4-4f70-a9df-b82f914e7b58/kube-rbac-proxy/0.log"
Jul 17 22:43:38.936013 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:38.935987    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zq9m4_48f183d4-25b4-4f70-a9df-b82f914e7b58/init-textfile/0.log"
Jul 17 22:43:39.086821 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:39.086703    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d9c621c2-d1a4-4b88-83c4-6d6aae5750b6/prometheus/0.log"
Jul 17 22:43:39.110419 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:39.110385    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d9c621c2-d1a4-4b88-83c4-6d6aae5750b6/config-reloader/0.log"
Jul 17 22:43:39.135154 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:39.135097    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d9c621c2-d1a4-4b88-83c4-6d6aae5750b6/thanos-sidecar/0.log"
Jul 17 22:43:39.161533 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:39.161507    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d9c621c2-d1a4-4b88-83c4-6d6aae5750b6/kube-rbac-proxy-web/0.log"
Jul 17 22:43:39.189845 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:39.189818    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d9c621c2-d1a4-4b88-83c4-6d6aae5750b6/kube-rbac-proxy/0.log"
Jul 17 22:43:39.218685 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:39.218657    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d9c621c2-d1a4-4b88-83c4-6d6aae5750b6/kube-rbac-proxy-thanos/0.log"
Jul 17 22:43:39.249670 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:39.249637    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d9c621c2-d1a4-4b88-83c4-6d6aae5750b6/init-config-reloader/0.log"
Jul 17 22:43:40.850259 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:40.850232    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-22npp_13fc622a-3c51-485a-8b45-1ee687ceeb17/networking-console-plugin/0.log"
Jul 17 22:43:41.422569 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.422531    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"]
Jul 17 22:43:41.422950 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.422932    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e634b5ee-b530-4c81-afb1-15f718eaace2" containerName="revoke-keys"
Jul 17 22:43:41.423053 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.422953    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e634b5ee-b530-4c81-afb1-15f718eaace2" containerName="revoke-keys"
Jul 17 22:43:41.423053 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.422968    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c074c4d3-ba15-4058-a0b4-812e4e543e10" containerName="revoke-keys"
Jul 17 22:43:41.423053 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.422973    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c074c4d3-ba15-4058-a0b4-812e4e543e10" containerName="revoke-keys"
Jul 17 22:43:41.423053 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.422978    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e634b5ee-b530-4c81-afb1-15f718eaace2" containerName="revoke-keys"
Jul 17 22:43:41.423053 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.422985    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e634b5ee-b530-4c81-afb1-15f718eaace2" containerName="revoke-keys"
Jul 17 22:43:41.423053 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.422998    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e634b5ee-b530-4c81-afb1-15f718eaace2" containerName="revoke-keys"
Jul 17 22:43:41.423053 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.423003    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e634b5ee-b530-4c81-afb1-15f718eaace2" containerName="revoke-keys"
Jul 17 22:43:41.423610 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.423090    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e634b5ee-b530-4c81-afb1-15f718eaace2" containerName="revoke-keys"
Jul 17 22:43:41.423610 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.423100    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e634b5ee-b530-4c81-afb1-15f718eaace2" containerName="revoke-keys"
Jul 17 22:43:41.423610 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.423107    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed6c2901-7a95-49d9-aba6-5315349636ab" containerName="revoke-keys"
Jul 17 22:43:41.427819 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.427801    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.430363 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.430341    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-728z5\"/\"openshift-service-ca.crt\""
Jul 17 22:43:41.430646 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.430619    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-728z5\"/\"kube-root-ca.crt\""
Jul 17 22:43:41.432297 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.432275    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-728z5\"/\"default-dockercfg-jpndz\""
Jul 17 22:43:41.435911 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.435891    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"]
Jul 17 22:43:41.513607 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.513570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d928e7c-b720-448b-a9a5-ce606208f17b-podres\") pod \"perf-node-gather-daemonset-v572j\" (UID: \"2d928e7c-b720-448b-a9a5-ce606208f17b\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.513807 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.513619    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d928e7c-b720-448b-a9a5-ce606208f17b-proc\") pod \"perf-node-gather-daemonset-v572j\" (UID: \"2d928e7c-b720-448b-a9a5-ce606208f17b\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.513807 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.513691    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d928e7c-b720-448b-a9a5-ce606208f17b-lib-modules\") pod \"perf-node-gather-daemonset-v572j\" (UID: \"2d928e7c-b720-448b-a9a5-ce606208f17b\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.513807 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.513735    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdl5d\" (UniqueName: \"kubernetes.io/projected/2d928e7c-b720-448b-a9a5-ce606208f17b-kube-api-access-gdl5d\") pod \"perf-node-gather-daemonset-v572j\" (UID: \"2d928e7c-b720-448b-a9a5-ce606208f17b\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.513807 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.513765    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d928e7c-b720-448b-a9a5-ce606208f17b-sys\") pod \"perf-node-gather-daemonset-v572j\" (UID: \"2d928e7c-b720-448b-a9a5-ce606208f17b\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.614544 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.614509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d928e7c-b720-448b-a9a5-ce606208f17b-lib-modules\") pod \"perf-node-gather-daemonset-v572j\" (UID: \"2d928e7c-b720-448b-a9a5-ce606208f17b\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.614722 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.614555    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdl5d\" (UniqueName: \"kubernetes.io/projected/2d928e7c-b720-448b-a9a5-ce606208f17b-kube-api-access-gdl5d\") pod \"perf-node-gather-daemonset-v572j\" (UID: \"2d928e7c-b720-448b-a9a5-ce606208f17b\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.614722 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.614595    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d928e7c-b720-448b-a9a5-ce606208f17b-sys\") pod \"perf-node-gather-daemonset-v572j\" (UID: \"2d928e7c-b720-448b-a9a5-ce606208f17b\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.614722 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.614660    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d928e7c-b720-448b-a9a5-ce606208f17b-podres\") pod \"perf-node-gather-daemonset-v572j\" (UID: \"2d928e7c-b720-448b-a9a5-ce606208f17b\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.614722 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.614702    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d928e7c-b720-448b-a9a5-ce606208f17b-proc\") pod \"perf-node-gather-daemonset-v572j\" (UID: \"2d928e7c-b720-448b-a9a5-ce606208f17b\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.614722 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.614706    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d928e7c-b720-448b-a9a5-ce606208f17b-lib-modules\") pod \"perf-node-gather-daemonset-v572j\" (UID: \"2d928e7c-b720-448b-a9a5-ce606208f17b\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.614978 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.614755    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d928e7c-b720-448b-a9a5-ce606208f17b-sys\") pod \"perf-node-gather-daemonset-v572j\" (UID: \"2d928e7c-b720-448b-a9a5-ce606208f17b\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.614978 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.614778    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d928e7c-b720-448b-a9a5-ce606208f17b-proc\") pod \"perf-node-gather-daemonset-v572j\" (UID: \"2d928e7c-b720-448b-a9a5-ce606208f17b\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.614978 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.614790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d928e7c-b720-448b-a9a5-ce606208f17b-podres\") pod \"perf-node-gather-daemonset-v572j\" (UID: \"2d928e7c-b720-448b-a9a5-ce606208f17b\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.623400 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.623367    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdl5d\" (UniqueName: \"kubernetes.io/projected/2d928e7c-b720-448b-a9a5-ce606208f17b-kube-api-access-gdl5d\") pod \"perf-node-gather-daemonset-v572j\" (UID: \"2d928e7c-b720-448b-a9a5-ce606208f17b\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.738906 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.738808    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:41.874204 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:41.874178    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"]
Jul 17 22:43:41.876176 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:43:41.876149    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2d928e7c_b720_448b_a9a5_ce606208f17b.slice/crio-b286df1c20ab5b2e7c6b92f23c891b83e7a84783ef41620ab9b6ade186220465 WatchSource:0}: Error finding container b286df1c20ab5b2e7c6b92f23c891b83e7a84783ef41620ab9b6ade186220465: Status 404 returned error can't find the container with id b286df1c20ab5b2e7c6b92f23c891b83e7a84783ef41620ab9b6ade186220465
Jul 17 22:43:42.125833 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:42.125805    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-568f5ff577-c6zrn_43df38e3-f93b-4b13-8fb0-29359252f3ff/console/0.log"
Jul 17 22:43:42.167195 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:42.167164    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-8p6dp_7929fd6d-36e0-46bf-83e6-7a93999d2f01/download-server/0.log"
Jul 17 22:43:42.228749 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:42.228711    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j" event={"ID":"2d928e7c-b720-448b-a9a5-ce606208f17b","Type":"ContainerStarted","Data":"60c825188a68f685cab5e17db1ad8f47d26b2a5e4fa275ec239aac645a08b0f5"}
Jul 17 22:43:42.228960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:42.228772    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j" event={"ID":"2d928e7c-b720-448b-a9a5-ce606208f17b","Type":"ContainerStarted","Data":"b286df1c20ab5b2e7c6b92f23c891b83e7a84783ef41620ab9b6ade186220465"}
Jul 17 22:43:42.228960 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:42.228816    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:42.245273 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:42.245221    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j" podStartSLOduration=1.245203537 podStartE2EDuration="1.245203537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 22:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:43:42.243509042 +0000 UTC m=+3485.632525714" watchObservedRunningTime="2026-07-17 22:43:42.245203537 +0000 UTC m=+3485.634220179"
Jul 17 22:43:43.182878 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:43.182838    2565 scope.go:117] "RemoveContainer" containerID="1b7656c5c97b7667921d70e796d81430b09a45020263dfcd98e1a72e8450dec1"
Jul 17 22:43:43.878428 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:43.878399    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8hzbf_13be2284-1e6e-414b-bd17-01caea2dc51c/dns/0.log"
Jul 17 22:43:43.898761 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:43.898735    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8hzbf_13be2284-1e6e-414b-bd17-01caea2dc51c/kube-rbac-proxy/0.log"
Jul 17 22:43:44.025107 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:44.025019    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sq8rg_79e9c71a-027c-4087-a6e0-1ecf5753a7b5/dns-node-resolver/0.log"
Jul 17 22:43:44.240921 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:44.240887    2565 generic.go:356] "Generic (PLEG): container finished" podID="19035086-3bce-4efb-b040-d5305b4d324f" containerID="d8df5b9adaf04bf7258190d9722ef561d23bb0909974f5003df6043f43cb75fe" exitCode=6
Jul 17 22:43:44.241337 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:44.240955    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q" event={"ID":"19035086-3bce-4efb-b040-d5305b4d324f","Type":"ContainerDied","Data":"d8df5b9adaf04bf7258190d9722ef561d23bb0909974f5003df6043f43cb75fe"}
Jul 17 22:43:44.241337 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:44.241017    2565 scope.go:117] "RemoveContainer" containerID="1b7656c5c97b7667921d70e796d81430b09a45020263dfcd98e1a72e8450dec1"
Jul 17 22:43:44.241337 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:44.241297    2565 scope.go:117] "RemoveContainer" containerID="d8df5b9adaf04bf7258190d9722ef561d23bb0909974f5003df6043f43cb75fe"
Jul 17 22:43:44.241577 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:43:44.241554    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q_odh-ai-gateway-infra(19035086-3bce-4efb-b040-d5305b4d324f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q" podUID="19035086-3bce-4efb-b040-d5305b4d324f"
Jul 17 22:43:44.570464 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:44.570432    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-55498b58b5-9kpmt_f8504435-3631-4552-8ed1-70376a9ff7b4/registry/0.log"
Jul 17 22:43:44.624879 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:44.624837    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ls9f6_b77bccab-4a6d-4e18-8206-9ad86e070317/node-ca/0.log"
Jul 17 22:43:45.269464 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:45.269413    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q"]
Jul 17 22:43:45.449391 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:45.449367    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q"
Jul 17 22:43:45.546304 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:45.546208    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19035086-3bce-4efb-b040-d5305b4d324f-maas-api-service-ca\") pod \"19035086-3bce-4efb-b040-d5305b4d324f\" (UID: \"19035086-3bce-4efb-b040-d5305b4d324f\") "
Jul 17 22:43:45.546603 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:45.546580    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19035086-3bce-4efb-b040-d5305b4d324f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "19035086-3bce-4efb-b040-d5305b4d324f" (UID: "19035086-3bce-4efb-b040-d5305b4d324f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:43:45.647595 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:45.647557    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19035086-3bce-4efb-b040-d5305b4d324f-maas-api-service-ca\") on node \"ip-10-0-129-11.ec2.internal\" DevicePath \"\""
Jul 17 22:43:45.697533 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:45.697508    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6782hq7p_314b915f-30de-4d8d-af9e-01f8d2b7fd08/istio-proxy/0.log"
Jul 17 22:43:45.821768 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:45.821663    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-8l8sd_b33878cc-2675-4a04-ad0d-7a2cf0d4db99/discovery/0.log"
Jul 17 22:43:45.866997 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:45.866965    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85fdd95b7f-x9zvr_81716365-9c9f-4819-b6d4-e948453cb094/kube-auth-proxy/0.log"
Jul 17 22:43:45.895263 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:45.895229    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-pvfz2_fc686a0b-25d0-4d64-bf27-ff5ee005090b/istio-proxy/0.log"
Jul 17 22:43:45.924324 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:45.924291    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-pvfz2_fc686a0b-25d0-4d64-bf27-ff5ee005090b/istio-proxy/1.log"
Jul 17 22:43:46.254160 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:46.254136    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q"
Jul 17 22:43:46.254322 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:46.254179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q" event={"ID":"19035086-3bce-4efb-b040-d5305b4d324f","Type":"ContainerDied","Data":"0a6921fd3c35b784041b5f8453b533e94cfa2ef01b92de3d36b8e1623af39980"}
Jul 17 22:43:46.254322 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:46.254243    2565 scope.go:117] "RemoveContainer" containerID="d8df5b9adaf04bf7258190d9722ef561d23bb0909974f5003df6043f43cb75fe"
Jul 17 22:43:46.276622 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:46.276596    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q"]
Jul 17 22:43:46.289475 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:46.289449    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-8vr2q"]
Jul 17 22:43:46.816647 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:46.816620    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q5vtj_2c961b41-034e-40be-a3f6-292e81593596/serve-healthcheck-canary/0.log"
Jul 17 22:43:47.188788 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.188695    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19035086-3bce-4efb-b040-d5305b4d324f" path="/var/lib/kubelet/pods/19035086-3bce-4efb-b040-d5305b4d324f/volumes"
Jul 17 22:43:47.189094 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.189047    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-nllz2"]
Jul 17 22:43:47.189433 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.189417    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19035086-3bce-4efb-b040-d5305b4d324f" containerName="revoke-keys"
Jul 17 22:43:47.189509 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.189436    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="19035086-3bce-4efb-b040-d5305b4d324f" containerName="revoke-keys"
Jul 17 22:43:47.189509 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.189463    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19035086-3bce-4efb-b040-d5305b4d324f" containerName="revoke-keys"
Jul 17 22:43:47.189509 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.189471    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="19035086-3bce-4efb-b040-d5305b4d324f" containerName="revoke-keys"
Jul 17 22:43:47.189673 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.189572    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="19035086-3bce-4efb-b040-d5305b4d324f" containerName="revoke-keys"
Jul 17 22:43:47.189673 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.189585    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="19035086-3bce-4efb-b040-d5305b4d324f" containerName="revoke-keys"
Jul 17 22:43:47.189673 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.189596    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e634b5ee-b530-4c81-afb1-15f718eaace2" containerName="revoke-keys"
Jul 17 22:43:47.194492 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.194391    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-nllz2"
Jul 17 22:43:47.196665 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.196632    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-nllz2"]
Jul 17 22:43:47.197043 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.197026    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:43:47.197563 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.197547    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\""
Jul 17 22:43:47.264188 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.264152    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd0ef0a-fc5a-473f-834f-326dc1207289-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-nllz2\" (UID: \"fdd0ef0a-fc5a-473f-834f-326dc1207289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-nllz2"
Jul 17 22:43:47.365528 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.365490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd0ef0a-fc5a-473f-834f-326dc1207289-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-nllz2\" (UID: \"fdd0ef0a-fc5a-473f-834f-326dc1207289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-nllz2"
Jul 17 22:43:47.366311 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.366286    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd0ef0a-fc5a-473f-834f-326dc1207289-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-nllz2\" (UID: \"fdd0ef0a-fc5a-473f-834f-326dc1207289\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-nllz2"
Jul 17 22:43:47.507383 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.507343    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-nllz2"
Jul 17 22:43:47.604375 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.604345    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdvns_24d59108-63d1-4002-9de4-8cf8287e3992/kube-rbac-proxy/0.log"
Jul 17 22:43:47.629837 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.629809    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdvns_24d59108-63d1-4002-9de4-8cf8287e3992/exporter/0.log"
Jul 17 22:43:47.646636 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.646606    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-nllz2"]
Jul 17 22:43:47.649894 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:43:47.649867    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd0ef0a_fc5a_473f_834f_326dc1207289.slice/crio-1adacc10c9f2749a9fe1144dd0370d9b670b277db391e4e808fc32a50ef8a44e WatchSource:0}: Error finding container 1adacc10c9f2749a9fe1144dd0370d9b670b277db391e4e808fc32a50ef8a44e: Status 404 returned error can't find the container with id 1adacc10c9f2749a9fe1144dd0370d9b670b277db391e4e808fc32a50ef8a44e
Jul 17 22:43:47.652880 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:47.652856    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdvns_24d59108-63d1-4002-9de4-8cf8287e3992/extractor/0.log"
Jul 17 22:43:48.243684 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:48.243604    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-728z5/perf-node-gather-daemonset-v572j"
Jul 17 22:43:48.267122 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:48.267094    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-nllz2" event={"ID":"fdd0ef0a-fc5a-473f-834f-326dc1207289","Type":"ContainerStarted","Data":"f79efe4d4da750eb8325fd472a1562d2deb5b5bcd0e7bf8e6cf5d27056519efb"}
Jul 17 22:43:48.267269 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:48.267146    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-nllz2" event={"ID":"fdd0ef0a-fc5a-473f-834f-326dc1207289","Type":"ContainerStarted","Data":"1adacc10c9f2749a9fe1144dd0370d9b670b277db391e4e808fc32a50ef8a44e"}
Jul 17 22:43:48.312732 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:48.312679    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-nllz2" podStartSLOduration=1.31266112 podStartE2EDuration="1.31266112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:43:48.311586041 +0000 UTC m=+3491.700602694" watchObservedRunningTime="2026-07-17 22:43:48.31266112 +0000 UTC m=+3491.701677771"
Jul 17 22:43:49.273973 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:49.273938    2565 generic.go:356] "Generic (PLEG): container finished" podID="fdd0ef0a-fc5a-473f-834f-326dc1207289" containerID="f79efe4d4da750eb8325fd472a1562d2deb5b5bcd0e7bf8e6cf5d27056519efb" exitCode=6
Jul 17 22:43:49.274419 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:49.274035    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-nllz2" event={"ID":"fdd0ef0a-fc5a-473f-834f-326dc1207289","Type":"ContainerDied","Data":"f79efe4d4da750eb8325fd472a1562d2deb5b5bcd0e7bf8e6cf5d27056519efb"}
Jul 17 22:43:49.274419 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:49.274257    2565 scope.go:117] "RemoveContainer" containerID="f79efe4d4da750eb8325fd472a1562d2deb5b5bcd0e7bf8e6cf5d27056519efb"
Jul 17 22:43:49.975397 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:49.975363    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-r56kt_1416461e-cf9d-4168-aca9-b496c64bcf7c/manager/0.log"
Jul 17 22:43:50.119266 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:50.119174    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5fl2x_d2089a42-8dbb-413a-b6cf-749899b408c0/manager/0.log"
Jul 17 22:43:50.281045 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:50.281010    2565 generic.go:356] "Generic (PLEG): container finished" podID="fdd0ef0a-fc5a-473f-834f-326dc1207289" containerID="91e62c7182b6002b89a75a65380ab87f5c8a28043236e02255543d233e086b60" exitCode=6
Jul 17 22:43:50.281478 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:50.281047    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-nllz2" event={"ID":"fdd0ef0a-fc5a-473f-834f-326dc1207289","Type":"ContainerDied","Data":"91e62c7182b6002b89a75a65380ab87f5c8a28043236e02255543d233e086b60"}
Jul 17 22:43:50.281478 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:50.281117    2565 scope.go:117] "RemoveContainer" containerID="f79efe4d4da750eb8325fd472a1562d2deb5b5bcd0e7bf8e6cf5d27056519efb"
Jul 17 22:43:50.281818 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:50.281713    2565 scope.go:117] "RemoveContainer" containerID="91e62c7182b6002b89a75a65380ab87f5c8a28043236e02255543d233e086b60"
Jul 17 22:43:50.282196 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:43:50.282160    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-nllz2_odh-ai-gateway-infra(fdd0ef0a-fc5a-473f-834f-326dc1207289)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-nllz2" podUID="fdd0ef0a-fc5a-473f-834f-326dc1207289"
Jul 17 22:43:50.362308 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:50.362281    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-z8r8m_877e02a7-c53c-4331-b7d0-409aa961ec39/manager/0.log"
Jul 17 22:43:50.557950 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:50.557920    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bff9999cf-xpc64_eec8dce4-acea-42e6-add7-879b08ce7aa8/manager/0.log"
Jul 17 22:43:50.578392 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:50.578372    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bff9999cf-xpc64_eec8dce4-acea-42e6-add7-879b08ce7aa8/copy-manifests/0.log"
Jul 17 22:43:51.288152 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:51.288127    2565 scope.go:117] "RemoveContainer" containerID="91e62c7182b6002b89a75a65380ab87f5c8a28043236e02255543d233e086b60"
Jul 17 22:43:51.288528 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:43:51.288338    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-nllz2_odh-ai-gateway-infra(fdd0ef0a-fc5a-473f-834f-326dc1207289)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-nllz2" podUID="fdd0ef0a-fc5a-473f-834f-326dc1207289"
Jul 17 22:43:51.992374 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:51.992294    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6782hq7p_314b915f-30de-4d8d-af9e-01f8d2b7fd08/istio-proxy/0.log"
Jul 17 22:43:52.101752 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:52.101717    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-8l8sd_b33878cc-2675-4a04-ad0d-7a2cf0d4db99/discovery/0.log"
Jul 17 22:43:52.131365 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:52.131337    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85fdd95b7f-x9zvr_81716365-9c9f-4819-b6d4-e948453cb094/kube-auth-proxy/0.log"
Jul 17 22:43:52.151974 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:52.151940    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-pvfz2_fc686a0b-25d0-4d64-bf27-ff5ee005090b/istio-proxy/0.log"
Jul 17 22:43:52.182095 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:52.182051    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-pvfz2_fc686a0b-25d0-4d64-bf27-ff5ee005090b/istio-proxy/1.log"
Jul 17 22:43:53.508941 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:53.508911    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6782hq7p_314b915f-30de-4d8d-af9e-01f8d2b7fd08/istio-proxy/0.log"
Jul 17 22:43:53.623542 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:53.623496    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-8l8sd_b33878cc-2675-4a04-ad0d-7a2cf0d4db99/discovery/0.log"
Jul 17 22:43:53.653010 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:53.652984    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85fdd95b7f-x9zvr_81716365-9c9f-4819-b6d4-e948453cb094/kube-auth-proxy/0.log"
Jul 17 22:43:53.681233 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:53.681203    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-pvfz2_fc686a0b-25d0-4d64-bf27-ff5ee005090b/istio-proxy/0.log"
Jul 17 22:43:53.709899 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:53.709866    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-pvfz2_fc686a0b-25d0-4d64-bf27-ff5ee005090b/istio-proxy/1.log"
Jul 17 22:43:55.630563 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:55.630533    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk"]
Jul 17 22:43:55.630972 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:55.630957    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19035086-3bce-4efb-b040-d5305b4d324f" containerName="revoke-keys"
Jul 17 22:43:55.631022 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:55.630974    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="19035086-3bce-4efb-b040-d5305b4d324f" containerName="revoke-keys"
Jul 17 22:43:55.631095 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:55.631082    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="19035086-3bce-4efb-b040-d5305b4d324f" containerName="revoke-keys"
Jul 17 22:43:55.639240 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:55.639213    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk"
Jul 17 22:43:55.644122 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:55.644089    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk"]
Jul 17 22:43:55.743937 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:55.743895    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4ee72b-f066-4ed5-ac09-6d8d2a8669cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk\" (UID: \"be4ee72b-f066-4ed5-ac09-6d8d2a8669cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk"
Jul 17 22:43:55.844888 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:55.844845    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4ee72b-f066-4ed5-ac09-6d8d2a8669cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk\" (UID: \"be4ee72b-f066-4ed5-ac09-6d8d2a8669cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk"
Jul 17 22:43:55.845549 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:55.845527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4ee72b-f066-4ed5-ac09-6d8d2a8669cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk\" (UID: \"be4ee72b-f066-4ed5-ac09-6d8d2a8669cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk"
Jul 17 22:43:55.952192 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:55.952094    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk"
Jul 17 22:43:56.117134 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:56.117105    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk"]
Jul 17 22:43:56.119791 ip-10-0-129-11 kubenswrapper[2565]: W0717 22:43:56.119761    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4ee72b_f066_4ed5_ac09_6d8d2a8669cc.slice/crio-4cc205b4d5f1163378243d123ca70c13f1ada6177df898dcc6a125dcceea7e0e WatchSource:0}: Error finding container 4cc205b4d5f1163378243d123ca70c13f1ada6177df898dcc6a125dcceea7e0e: Status 404 returned error can't find the container with id 4cc205b4d5f1163378243d123ca70c13f1ada6177df898dcc6a125dcceea7e0e
Jul 17 22:43:56.312498 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:56.312465    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk" event={"ID":"be4ee72b-f066-4ed5-ac09-6d8d2a8669cc","Type":"ContainerStarted","Data":"4cc205b4d5f1163378243d123ca70c13f1ada6177df898dcc6a125dcceea7e0e"}
Jul 17 22:43:57.165279 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:57.165199    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-4ll6f_75819c53-b542-4a7c-af75-49270bed8d13/migrator/0.log"
Jul 17 22:43:57.188854 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:57.188825    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-4ll6f_75819c53-b542-4a7c-af75-49270bed8d13/graceful-termination/0.log"
Jul 17 22:43:57.318745 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:57.318710    2565 generic.go:356] "Generic (PLEG): container finished" podID="be4ee72b-f066-4ed5-ac09-6d8d2a8669cc" containerID="d187f019d68908c331b7d4e8433d77150d6db02fffa579a4e1657a73b03c4abe" exitCode=6
Jul 17 22:43:57.318932 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:57.318756    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk" event={"ID":"be4ee72b-f066-4ed5-ac09-6d8d2a8669cc","Type":"ContainerDied","Data":"d187f019d68908c331b7d4e8433d77150d6db02fffa579a4e1657a73b03c4abe"}
Jul 17 22:43:57.318980 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:57.318938    2565 scope.go:117] "RemoveContainer" containerID="d187f019d68908c331b7d4e8433d77150d6db02fffa579a4e1657a73b03c4abe"
Jul 17 22:43:58.326112 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:58.326057    2565 generic.go:356] "Generic (PLEG): container finished" podID="be4ee72b-f066-4ed5-ac09-6d8d2a8669cc" containerID="091547d5ad9e6c6e9d3399c9ca85c3b131ee4f22925f6e8d75b372fbb9bb232e" exitCode=6
Jul 17 22:43:58.326523 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:58.326135    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk" event={"ID":"be4ee72b-f066-4ed5-ac09-6d8d2a8669cc","Type":"ContainerDied","Data":"091547d5ad9e6c6e9d3399c9ca85c3b131ee4f22925f6e8d75b372fbb9bb232e"}
Jul 17 22:43:58.326523 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:58.326199    2565 scope.go:117] "RemoveContainer" containerID="d187f019d68908c331b7d4e8433d77150d6db02fffa579a4e1657a73b03c4abe"
Jul 17 22:43:58.326523 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:58.326415    2565 scope.go:117] "RemoveContainer" containerID="091547d5ad9e6c6e9d3399c9ca85c3b131ee4f22925f6e8d75b372fbb9bb232e"
Jul 17 22:43:58.326686 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:43:58.326661    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk_odh-ai-gateway-infra(be4ee72b-f066-4ed5-ac09-6d8d2a8669cc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk" podUID="be4ee72b-f066-4ed5-ac09-6d8d2a8669cc"
Jul 17 22:43:58.898164 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:58.898133    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2lwh_d604d7d6-843e-47f3-b837-31545356dbf3/kube-multus-additional-cni-plugins/0.log"
Jul 17 22:43:58.926376 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:58.926344    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2lwh_d604d7d6-843e-47f3-b837-31545356dbf3/egress-router-binary-copy/0.log"
Jul 17 22:43:58.957093 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:58.957041    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2lwh_d604d7d6-843e-47f3-b837-31545356dbf3/cni-plugins/0.log"
Jul 17 22:43:58.982605 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:58.982541    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2lwh_d604d7d6-843e-47f3-b837-31545356dbf3/bond-cni-plugin/0.log"
Jul 17 22:43:59.004963 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:59.004935    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2lwh_d604d7d6-843e-47f3-b837-31545356dbf3/routeoverride-cni/0.log"
Jul 17 22:43:59.033602 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:59.033572    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2lwh_d604d7d6-843e-47f3-b837-31545356dbf3/whereabouts-cni-bincopy/0.log"
Jul 17 22:43:59.058609 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:59.058580    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h2lwh_d604d7d6-843e-47f3-b837-31545356dbf3/whereabouts-cni/0.log"
Jul 17 22:43:59.333283 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:59.333254    2565 scope.go:117] "RemoveContainer" containerID="091547d5ad9e6c6e9d3399c9ca85c3b131ee4f22925f6e8d75b372fbb9bb232e"
Jul 17 22:43:59.333761 ip-10-0-129-11 kubenswrapper[2565]: E0717 22:43:59.333498    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk_odh-ai-gateway-infra(be4ee72b-f066-4ed5-ac09-6d8d2a8669cc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-ln6qk" podUID="be4ee72b-f066-4ed5-ac09-6d8d2a8669cc"
Jul 17 22:43:59.350111 ip-10-0-129-11 kubenswrapper[2565]: I0717 22:43:59.350058    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-p2d44_a05b7f74-b162-43d1-b600-53c5904268de/kube-multus/0.log"